public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 6/6] Check removal of in-reading segment file.
36+ messages / 9 participants
[nested] [flat]
* [PATCH 6/6] Check removal of in-reading segment file.
@ 2018-10-26 01:07 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-26 01:07 UTC (permalink / raw)
Checkpoints can recycle a segment file while it is being read by
ReadRecord and that leads to an apparently odd error message during
logical decoding. This patch explicitly checks that then error out
immediately. Reading a recycled file is safe. Inconsistency caused by
overwrites as a new segment are caught by page/record validation. So
this is only for keeping consistency with the wal_status shown in
pg_replication_slots.
---
src/backend/access/transam/xlogreader.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 33ccfc1553..4999892932 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -27,6 +27,7 @@
#ifndef FRONTEND
#include "miscadmin.h"
+#include "access/xlog.h"
#include "utils/memutils.h"
#endif
@@ -225,7 +226,9 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
uint32 pageHeaderSize;
bool gotheader;
int readOff;
-
+#ifndef FRONTEND
+ XLogSegNo targetSegNo;
+#endif
/*
* randAccess indicates whether to verify the previous-record pointer of
* the record we're reading. We only do this if we're reading
@@ -271,6 +274,22 @@ XLogReadRecord(XLogReaderState *state, XLogRecPtr RecPtr, char **errormsg)
targetPagePtr = RecPtr - (RecPtr % XLOG_BLCKSZ);
targetRecOff = RecPtr % XLOG_BLCKSZ;
+#ifndef FRONTEND
+ /*
+ * Although It's safe that the current segment is recycled as a new
+ * segment since we check the page/record header at reading, it leads to
+ * an apparently strange error message when logical replication, which can
+ * be prevented by explicitly checking if the current segment is removed.
+ */
+ XLByteToSeg(targetPagePtr, targetSegNo, state->wal_segment_size);
+ if (targetSegNo <= XLogGetLastRemovedSegno())
+ {
+ report_invalid_record(state,
+ "WAL segment for LSN %X/%X has been removed",
+ (uint32)(RecPtr >> 32), (uint32) RecPtr);
+ goto err;
+ }
+#endif
/*
* Read the page containing the record into state->readBuf. Request enough
* byte to cover the whole record header, or at least the part of it that
--
2.16.3
----Next_Part(Tue_Jul_30_21_30_45_2019_680)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
@ 2023-01-11 21:58 Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Justin Pryzby @ 2023-01-11 21:58 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Andres Freund <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>
> Subject: [PATCH v45 4/5] Add system view tracking IO ops per backend type
The patch can/will fail with:
CREATE TABLESPACE test_io_shared_stats_tblspc LOCATION '';
+WARNING: tablespaces created by regression test cases should have names starting with "regress_"
CREATE TABLESPACE test_stats LOCATION '';
+WARNING: tablespaces created by regression test cases should have names starting with "regress_"
(I already sent patches to address the omission in cirrus.yml)
1760 : errhint("Target must be \"archiver\", \"io\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
=> Do you want to put these in order?
pgstat_get_io_op_name() isn't currently being hit by tests; actually,
it's completely unused.
FlushRelationBuffers() isn't being hit for local buffers.
> + <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
> + <entry>
> + One row per backend type, context, target object combination showing
> + cluster-wide I/O statistics.
I suggest: "One row for each combination of of .."
> + The <structname>pg_stat_io</structname> and
> + <structname>pg_statio_</structname> set of views are especially useful for
> + determining the effectiveness of the buffer cache. When the number of actual
> + disk reads is much smaller than the number of buffer hits, then the cache is
> + satisfying most read requests without invoking a kernel call.
I would change this say "Postgres' own buffer cache is satisfying ..."
> However, these
> + statistics do not give the entire story: due to the way in which
> + <productname>PostgreSQL</productname> handles disk I/O, data that is not in
> + the <productname>PostgreSQL</productname> buffer cache might still reside in
> + the kernel's I/O cache, and might therefore still be fetched without
I suggest to refer to "the kernel's page cache"
> + The <structname>pg_stat_io</structname> view will contain one row for each
> + backend type, I/O context, and target I/O object combination showing
> + cluster-wide I/O statistics. Combinations which do not make sense are
> + omitted.
"..for each combination of .."
> + <varname>io_context</varname> for a type of I/O operation. For
"for I/O operations"
> + <literal>vacuum</literal>: I/O operations done outside of shared
> + buffers incurred while vacuuming and analyzing permanent relations.
s/incurred/performed/
> + <literal>bulkread</literal>: Qualifying large read I/O operations
> + done outside of shared buffers, for example, a sequential scan of a
> + large table.
I don't think it's correct to say that it's "outside of" shared-buffers.
s/Qualifying/Certain/
> + <literal>bulkwrite</literal>: Qualifying large write I/O operations
> + done outside of shared buffers, such as <command>COPY</command>.
Same
> + Target object of an I/O operation. Possible values are:
> + <itemizedlist>
> + <listitem>
> + <para>
> + <literal>relation</literal>: This includes permanent relations.
It says "includes permanent" but what seems to mean is that it
"exclusive of temporary relations".
> + <row>
> + <entry role="catalog_table_entry">
> + <para role="column_definition">
> + <structfield>read</structfield> <type>bigint</type>
> + </para>
> + <para>
> + Number of read operations in units of <varname>op_bytes</varname>.
This looks too much like it means "bytes".
Should say: "in number of blocks of size >op_bytes<"
But wait - is it the number of read operations "in units of op_bytes"
(which would means this already multiplied by op_bytes, and is in units
of bytes).
Or the "number of read operations" *of* op_bytes chunks ? Which would
mean this is a "pure" number, and could be multipled by op_bytes to
obtain a size in bytes.
> + Number of write operations in units of <varname>op_bytes</varname>.
> + Number of relation extend operations in units of
> + <varname>op_bytes</varname>.
same
> + In <varname>io_context</varname> <literal>normal</literal>, this counts
> + the number of times a block was evicted from a buffer and replaced with
> + another block. In <varname>io_context</varname>s
> + <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
> + <literal>vacuum</literal>, this counts the number of times a block was
> + evicted from shared buffers in order to add the shared buffer to a
> + separate size-limited ring buffer.
This never defines what "evicted" means. Does it mea that a dirty
buffer was written out ?
> + The number of times an existing buffer in a size-limited ring buffer
> + outside of shared buffers was reused as part of an I/O operation in the
> + <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
> + <literal>vacuum</literal> <varname>io_context</varname>s.
Maybe say "as part of a bulk I/O operation (bulkread, bulkwrite, or
vacuum)."
> + <para>
> + <structname>pg_stat_io</structname> can be used to inform database tuning.
> + For example:
> + <itemizedlist>
> + <listitem>
> + <para>
> + A high <varname>evicted</varname> count can indicate that shared buffers
> + should be increased.
> + </para>
> + </listitem>
> + <listitem>
> + <para>
> + Client backends rely on the checkpointer to ensure data is persisted to
> + permanent storage. Large numbers of <varname>files_synced</varname> by
> + <literal>client backend</literal>s could indicate a misconfiguration of
> + shared buffers or of checkpointer. More information on checkpointer
of *the* checkpointer
> + Normally, client backends should be able to rely on auxiliary processes
> + like the checkpointer and background writer to write out dirty data as
*the* bg writer
> + much as possible. Large numbers of writes by client backends could
> + indicate a misconfiguration of shared buffers or of checkpointer. More
*the* ckpointer
Should this link to various docs for checkpointer/bgwriter ?
Maybe the docs for ALTER/COPY/VACUUM/CREATE/etc should be updated to
refer to some central description of ring buffers. Maybe something
should be included to the appendix.
--
Justin
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
@ 2023-01-13 02:19 ` Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Melanie Plageman @ 2023-01-13 02:19 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Andres Freund <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; vignesh C <[email protected]>
Attached is v46.
On Wed, Dec 28, 2022 at 6:56 PM Andres Freund <[email protected]> wrote:
> On 2022-10-06 13:42:09 -0400, Melanie Plageman wrote:
> > > Additionally, some minor notes:
> > >
> > > - Since the stats are counting blocks, it would make sense to prefix the view columns with "blks_", and word them in the past tense (to match current style), i.e. "blks_written", "blks_read", "blks_extended", "blks_fsynced" (realistically one would combine this new view with other data e.g. from pg_stat_database or pg_stat_statements, which all use the "blks_" prefix, and stop using pg_stat_bgwriter for this which does not use such a prefix)
> >
> > I have changed the column names to be in the past tense.
>
> For a while I was convinced by the consistency argument (after Melanie
> pointing it out to me). But the more I look, the less convinced I am. The
> existing IO related stats in pg_stat_database, pg_stat_bgwriter aren't past
> tense, just the ones in pg_stat_statements. pg_stat_database uses past tense
> for tup_*, but not xact_*, deadlocks, checksum_failures etc.
>
> And even pg_stat_statements isn't consistent about it - otherwise it'd be
> 'planned' instead of 'plans', 'called' instead of 'calls' etc.
>
> I started to look at the naming "tense" issue again, after I got "confused"
> about "extended", because that somehow makes me think about more detailed
> stats or such, rather than files getting extended.
>
> ISTM that 'evictions', 'extends', 'fsyncs', 'reads', 'reuses', 'writes' are
> clearer than the past tense versions, and about as consistent with existing
> columns.
I have updated the column names to the above recommendation.
On Wed, Jan 11, 2023 at 11:32 AM vignesh C <[email protected]> wrote:
>
> For some reason cfbot is not able to apply this patch as in [1],
> please have a look and post an updated patch if required:
> === Applying patches on top of PostgreSQL commit ID
> 3c6fc58209f24b959ee18f5d19ef96403d08f15c ===
> === applying patch
> ./v45-0001-pgindent-and-some-manual-cleanup-in-pgstat-relat.patch
> patching file src/backend/storage/buffer/bufmgr.c
> patching file src/backend/storage/buffer/localbuf.c
> patching file src/backend/utils/activity/pgstat.c
> patching file src/backend/utils/activity/pgstat_relation.c
> patching file src/backend/utils/adt/pgstatfuncs.c
> patching file src/include/pgstat.h
> patching file src/include/utils/pgstat_internal.h
> === applying patch ./v45-0002-pgstat-Infrastructure-to-track-IO-operations.patch
> gpatch: **** Only garbage was found in the patch input.
>
> [1] - http://cfbot.cputube.org/patch_41_3272.log
>
This was an issue with cfbot that Thomas has now fixed as he describes
in [1].
On Wed, Jan 11, 2023 at 4:58 PM Justin Pryzby <[email protected]> wrote:
>
> > Subject: [PATCH v45 4/5] Add system view tracking IO ops per backend type
>
> The patch can/will fail with:
>
> CREATE TABLESPACE test_io_shared_stats_tblspc LOCATION '';
> +WARNING: tablespaces created by regression test cases should have names starting with "regress_"
>
> CREATE TABLESPACE test_stats LOCATION '';
> +WARNING: tablespaces created by regression test cases should have names starting with "regress_"
>
> (I already sent patches to address the omission in cirrus.yml)
Thanks. I've fixed this
I make a tablespace in amcheck -- are there recommendations for naming
tablespaces in contrib also?
>
> 1760 : errhint("Target must be \"archiver\", \"io\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
> => Do you want to put these in order?
Thanks. Fixed.
> pgstat_get_io_op_name() isn't currently being hit by tests; actually,
> it's completely unused.
Deleted it.
> FlushRelationBuffers() isn't being hit for local buffers.
I added a test.
> > + <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
> > + <entry>
> > + One row per backend type, context, target object combination showing
> > + cluster-wide I/O statistics.
>
> I suggest: "One row for each combination of of .."
I have made this change.
> > + The <structname>pg_stat_io</structname> and
> > + <structname>pg_statio_</structname> set of views are especially useful for
> > + determining the effectiveness of the buffer cache. When the number of actual
> > + disk reads is much smaller than the number of buffer hits, then the cache is
> > + satisfying most read requests without invoking a kernel call.
>
> I would change this say "Postgres' own buffer cache is satisfying ..."
So, this is existing copy to which I added the pg_stat_io view name and
re-flowed the indentation.
However, I think your suggestions are a good idea, so I've taken them
and just rewritten this paragraph altogether.
>
> > However, these
> > + statistics do not give the entire story: due to the way in which
> > + <productname>PostgreSQL</productname> handles disk I/O, data that is not in
> > + the <productname>PostgreSQL</productname> buffer cache might still reside in
> > + the kernel's I/O cache, and might therefore still be fetched without
>
> I suggest to refer to "the kernel's page cache"
same applies here.
>
> > + The <structname>pg_stat_io</structname> view will contain one row for each
> > + backend type, I/O context, and target I/O object combination showing
> > + cluster-wide I/O statistics. Combinations which do not make sense are
> > + omitted.
>
> "..for each combination of .."
I have changed this.
>
> > + <varname>io_context</varname> for a type of I/O operation. For
>
> "for I/O operations"
So I actually mean for a type of I/O operation -- that is, relation data
is normally written to a shared buffer but sometimes we bypass shared
buffers and just call write and sometimes we use a buffer access
strategy and write it to a special ring buffer (made up of buffers
stolen from shared buffers, but still). So I don't want to say "for I/O
operations" because I think that would imply that writes of relation
data will always be in the same IO Context.
>
> > + <literal>vacuum</literal>: I/O operations done outside of shared
> > + buffers incurred while vacuuming and analyzing permanent relations.
>
> s/incurred/performed/
I changed this.
>
> > + <literal>bulkread</literal>: Qualifying large read I/O operations
> > + done outside of shared buffers, for example, a sequential scan of a
> > + large table.
>
> I don't think it's correct to say that it's "outside of" shared-buffers.
I suppose "outside of" gives the wrong idea. But I need to make clear
that this I/O is to and from buffers which are not a part of shared
buffers right now -- they may still be accessible from the same data
structures which access shared buffers but they are currently being used
in a different way.
> s/Qualifying/Certain/
I feel like qualifying is more specific than certain, but I would be open
to changing it if there was a specific reason you don't like it.
>
> > + <literal>bulkwrite</literal>: Qualifying large write I/O operations
> > + done outside of shared buffers, such as <command>COPY</command>.
>
> Same
>
> > + Target object of an I/O operation. Possible values are:
> > + <itemizedlist>
> > + <listitem>
> > + <para>
> > + <literal>relation</literal>: This includes permanent relations.
>
> It says "includes permanent" but what seems to mean is that it
> "exclusive of temporary relations".
I've changed this.
>
> > + <row>
> > + <entry role="catalog_table_entry">
> > + <para role="column_definition">
> > + <structfield>read</structfield> <type>bigint</type>
> > + </para>
> > + <para>
> > + Number of read operations in units of <varname>op_bytes</varname>.
>
> This looks too much like it means "bytes".
> Should say: "in number of blocks of size >op_bytes<"
>
> But wait - is it the number of read operations "in units of op_bytes"
> (which would means this already multiplied by op_bytes, and is in units
> of bytes).
>
> Or the "number of read operations" *of* op_bytes chunks ? Which would
> mean this is a "pure" number, and could be multipled by op_bytes to
> obtain a size in bytes.
It is the number of read operations of op_bytes size -- thanks so much
for pointing this out. The wording was really unclear.
The idea is that you can do something like:
SELECT pg_size_pretty(reads * op_bytes) FROM pg_stat_io;
and get it in bytes.
The view will contain other types of IO that are not in BLCKSZ chunks,
which is where this column will be handy.
>
> > + Number of write operations in units of <varname>op_bytes</varname>.
>
> > + Number of relation extend operations in units of
> > + <varname>op_bytes</varname>.
>
> same
>
> > + In <varname>io_context</varname> <literal>normal</literal>, this counts
> > + the number of times a block was evicted from a buffer and replaced with
> > + another block. In <varname>io_context</varname>s
> > + <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
> > + <literal>vacuum</literal>, this counts the number of times a block was
> > + evicted from shared buffers in order to add the shared buffer to a
> > + separate size-limited ring buffer.
>
> This never defines what "evicted" means. Does it mea that a dirty
> buffer was written out ?
Thanks. I've updated this.
>
> > + The number of times an existing buffer in a size-limited ring buffer
> > + outside of shared buffers was reused as part of an I/O operation in the
> > + <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
> > + <literal>vacuum</literal> <varname>io_context</varname>s.
>
> Maybe say "as part of a bulk I/O operation (bulkread, bulkwrite, or
> vacuum)."
I've changed this.
>
> > + <para>
> > + <structname>pg_stat_io</structname> can be used to inform database tuning.
>
> > + For example:
> > + <itemizedlist>
> > + <listitem>
> > + <para>
> > + A high <varname>evicted</varname> count can indicate that shared buffers
> > + should be increased.
> > + </para>
> > + </listitem>
> > + <listitem>
> > + <para>
> > + Client backends rely on the checkpointer to ensure data is persisted to
> > + permanent storage. Large numbers of <varname>files_synced</varname> by
> > + <literal>client backend</literal>s could indicate a misconfiguration of
> > + shared buffers or of checkpointer. More information on checkpointer
>
> of *the* checkpointer
>
> > + Normally, client backends should be able to rely on auxiliary processes
> > + like the checkpointer and background writer to write out dirty data as
>
> *the* bg writer
>
> > + much as possible. Large numbers of writes by client backends could
> > + indicate a misconfiguration of shared buffers or of checkpointer. More
>
> *the* ckpointer
I've made most of these changes.
> Should this link to various docs for checkpointer/bgwriter ?
I couldn't find docs related to tuning checkpointer outside of the WAL
configuration docs. There is the docs page for the CHECKPOINT command --
but I don't think that is very relevant here.
> Maybe the docs for ALTER/COPY/VACUUM/CREATE/etc should be updated to
> refer to some central description of ring buffers. Maybe something
> should be included to the appendix.
I agree it would be nice to explain Buffer Access Strategies in the docs.
- Melanie
[1] https://www.postgresql.org/message-id/CA%2BhUKGLiY1e%2B1%3DpB7hXJOyGj1dJOfgde%2BHmiSnv3gDKayUFJMA%40...
Attachments:
[text/x-patch] v46-0001-pgindent-and-some-manual-cleanup-in-pgstat-relat.patch (6.0K, ../../CAAKRu_bTjvHSd5tWWQkW8SWY-5frVp_vOJpFD+-V7jJG1Rf_Dw@mail.gmail.com/2-v46-0001-pgindent-and-some-manual-cleanup-in-pgstat-relat.patch)
download | inline diff:
From 8960d4b3902374a999b3c7e572995d70b2cb0557 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 9 Dec 2022 18:23:19 -0800
Subject: [PATCH v46 1/5] pgindent and some manual cleanup in pgstat related
code
---
src/backend/storage/buffer/bufmgr.c | 22 ++++++++++----------
src/backend/storage/buffer/localbuf.c | 4 ++--
src/backend/utils/activity/pgstat.c | 3 ++-
src/backend/utils/activity/pgstat_relation.c | 1 +
src/backend/utils/adt/pgstatfuncs.c | 2 +-
src/include/pgstat.h | 1 +
src/include/utils/pgstat_internal.h | 1 +
7 files changed, 19 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 3fb38a25cf..8075828e8a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -516,7 +516,7 @@ PrefetchSharedBuffer(SMgrRelation smgr_reln,
/* create a tag so we can lookup the buffer */
InitBufferTag(&newTag, &smgr_reln->smgr_rlocator.locator,
- forkNum, blockNum);
+ forkNum, blockNum);
/* determine its hash code and partition lock ID */
newHash = BufTableHashCode(&newTag);
@@ -3297,8 +3297,8 @@ DropRelationsAllBuffers(SMgrRelation *smgr_reln, int nlocators)
uint32 buf_state;
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (!use_bsearch)
@@ -3425,8 +3425,8 @@ DropDatabaseBuffers(Oid dbid)
uint32 buf_state;
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (bufHdr->tag.dbOid != dbid)
continue;
@@ -3572,8 +3572,8 @@ FlushRelationBuffers(Relation rel)
bufHdr = GetBufferDescriptor(i);
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (!BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator))
continue;
@@ -3645,8 +3645,8 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
uint32 buf_state;
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (!use_bsearch)
@@ -3880,8 +3880,8 @@ FlushDatabaseBuffers(Oid dbid)
bufHdr = GetBufferDescriptor(i);
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (bufHdr->tag.dbOid != dbid)
continue;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index b2720df6ea..8372acc383 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -610,8 +610,8 @@ AtProcExit_LocalBuffers(void)
{
/*
* We shouldn't be holding any remaining pins; if we are, and assertions
- * aren't enabled, we'll fail later in DropRelationBuffers while
- * trying to drop the temp rels.
+ * aren't enabled, we'll fail later in DropRelationBuffers while trying to
+ * drop the temp rels.
*/
CheckForLocalBufferLeaks();
}
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 7e9dc17e68..0fa5370bcd 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -426,7 +426,7 @@ pgstat_discard_stats(void)
ereport(DEBUG2,
(errcode_for_file_access(),
errmsg_internal("unlinked permanent statistics file \"%s\"",
- PGSTAT_STAT_PERMANENT_FILENAME)));
+ PGSTAT_STAT_PERMANENT_FILENAME)));
}
/*
@@ -986,6 +986,7 @@ pgstat_build_snapshot(void)
entry->data = MemoryContextAlloc(pgStatLocal.snapshot.context,
kind_info->shared_size);
+
/*
* Acquire the LWLock directly instead of using
* pg_stat_lock_entry_shared() which requires a reference.
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 1730425de1..2e20b93c20 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -783,6 +783,7 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
if (lstats->t_counts.t_numscans)
{
TimestampTz t = GetCurrentTransactionStopTimestamp();
+
if (t > tabentry->lastscan)
tabentry->lastscan = t;
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 6cddd74aa7..58bd1360b9 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -906,7 +906,7 @@ pg_stat_get_backend_client_addr(PG_FUNCTION_ARGS)
clean_ipv6_addr(beentry->st_clientaddr.addr.ss_family, remote_host);
PG_RETURN_DATUM(DirectFunctionCall1(inet_in,
- CStringGetDatum(remote_host)));
+ CStringGetDatum(remote_host)));
}
Datum
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index d3e965d744..5e3326a3b9 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -476,6 +476,7 @@ extern void pgstat_report_connect(Oid dboid);
extern PgStat_StatDBEntry *pgstat_fetch_stat_dbentry(Oid dboid);
+
/*
* Functions in pgstat_function.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 08412d6404..12fd51f1ae 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -626,6 +626,7 @@ extern void pgstat_wal_snapshot_cb(void);
extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
+
/*
* Functions in pgstat_xact.c
*/
--
2.34.1
[text/x-patch] v46-0005-pg_stat_io-documentation.patch (13.4K, ../../CAAKRu_bTjvHSd5tWWQkW8SWY-5frVp_vOJpFD+-V7jJG1Rf_Dw@mail.gmail.com/3-v46-0005-pg_stat_io-documentation.patch)
download | inline diff:
From 6792d761ebf5e9261dab452130dcc2cd5ac48d0e Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 9 Jan 2023 14:42:53 -0500
Subject: [PATCH v46 5/5] pg_stat_io documentation
Author: Melanie Plageman <[email protected]>
Author: Samay Sharma <[email protected]>
Reviewed-by: Maciek Sakrejda <[email protected]>
Reviewed-by: Lukas Fittl <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 318 +++++++++++++++++++++++++++++++++--
1 file changed, 304 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 8d51ca3773..12246d46f4 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -469,6 +469,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
+ <entry>
+ One row for each combination of backend type, context, and target object
+ containing cluster-wide I/O statistics.
+ See <link linkend="monitoring-pg-stat-io-view">
+ <structname>pg_stat_io</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_replication_slots</structname><indexterm><primary>pg_stat_replication_slots</primary></indexterm></entry>
<entry>One row per replication slot, showing statistics about the
@@ -665,20 +675,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</para>
<para>
- The <structname>pg_statio_</structname> views are primarily useful to
- determine the effectiveness of the buffer cache. When the number
- of actual disk reads is much smaller than the number of buffer
- hits, then the cache is satisfying most read requests without
- invoking a kernel call. However, these statistics do not give the
- entire story: due to the way in which <productname>PostgreSQL</productname>
- handles disk I/O, data that is not in the
- <productname>PostgreSQL</productname> buffer cache might still reside in the
- kernel's I/O cache, and might therefore still be fetched without
- requiring a physical read. Users interested in obtaining more
- detailed information on <productname>PostgreSQL</productname> I/O behavior are
- advised to use the <productname>PostgreSQL</productname> statistics views
- in combination with operating system utilities that allow insight
- into the kernel's handling of I/O.
+ The <structname>pg_stat_io</structname> and
+ <structname>pg_statio_</structname> set of views are useful for determining
+ the effectiveness of the buffer cache. They can be used to calculate a cache
+ hit ratio. Note that while <productname>PostgreSQL</productname>'s I/O
+ statistics capture most instances in which the kernel was invoked in order
+ to perform I/O, they do not differentiate between data which had to be
+ fetched from disk and that which already resided in the kernel page cache.
+ Users are advised to use the <productname>PostgreSQL</productname>
+ statistics views in combination with operating system utilities for a more
+ complete picture of their database's I/O performance.
</para>
</sect2>
@@ -3643,6 +3649,290 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_archived_wal</structfield> have also been successfully
archived.
</para>
+ </sect2>
+
+ <sect2 id="monitoring-pg-stat-io-view">
+ <title><structname>pg_stat_io</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_io</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_io</structname> view will contain one row for each
+ combination of backend type, I/O context, and target I/O object showing
+ cluster-wide I/O statistics. Combinations which do not make sense are
+ omitted.
+ </para>
+
+ <para>
+ Currently, I/O on relations (e.g. tables, indexes) is tracked. However,
+ relation I/O which bypasses shared buffers (e.g. when moving a table from one
+ tablespace to another) is currently not tracked.
+ </para>
+
+ <table id="pg-stat-io-view" xreflabel="pg_stat_io">
+ <title><structname>pg_stat_io</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>backend_type</structfield> <type>text</type>
+ </para>
+ <para>
+ Type of backend (e.g. background worker, autovacuum worker). See <link
+ linkend="monitoring-pg-stat-activity-view">
+ <structname>pg_stat_activity</structname></link> for more information
+ on <varname>backend_type</varname>s. Some
+ <varname>backend_type</varname>s do not accumulate I/O operation
+ statistics and will not be included in the view.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_context</structfield> <type>text</type>
+ </para>
+ <para>
+ The context of an I/O operation. Possible values are:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>normal</literal>: The default or standard
+ <varname>io_context</varname> for a type of I/O operation. For
+ example, by default, relation data is read into and written out from
+ shared buffers. Thus, reads and writes of relation data to and from
+ shared buffers are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>vacuum</literal>: I/O operations performed outside of shared
+ buffers while vacuuming and analyzing permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkread</literal>: Qualifying large read I/O operations
+ done outside of shared buffers, for example, a sequential scan of a
+ large table.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkwrite</literal>: Qualifying large write I/O operations
+ done outside of shared buffers, such as <command>COPY</command>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_object</structfield> <type>text</type>
+ </para>
+ <para>
+ Target object of an I/O operation. Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>relation</literal>: Permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>temp relation</literal>: Temporary relations.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>read</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of read operations of <varname>op_bytes</varname> size.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>written</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of write operations of <varname>op_bytes</varname> size.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>extended</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of relation extend operations of <varname>op_bytes</varname>
+ size.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>op_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of bytes per unit of I/O read, written, or extended.
+ </para>
+ <para>
+ Relation data reads, writes, and extends are done in
+ <varname>block_size</varname> units, derived from the build-time
+ parameter <symbol>BLCKSZ</symbol>, which is <literal>8192</literal> by
+ default.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>evicted</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a block has been written out from a shared or local
+ buffer in order to make it available for another use.
+ </para>
+ <para>
+ In <varname>io_context</varname> <literal>normal</literal>, this counts
+ the number of times a block was evicted from a buffer and replaced with
+ another block. In <varname>io_context</varname>s
+ <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
+ <literal>vacuum</literal>, this counts the number of times a block was
+ evicted from shared buffers in order to add the shared buffer to a
+ separate size-limited ring buffer for use in a bulk I/O operation.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reused</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of times an existing buffer in a size-limited ring buffer
+ outside of shared buffers was reused as part of an I/O operation in the
+ <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
+ <literal>vacuum</literal> <varname>io_context</varname>s.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>files_synced</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of <literal>fsync</literal> calls. These are only tracked in
+ <varname>io_context</varname> <literal>normal</literal>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset.
+ </para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Some <varname>backend_type</varname>s never perform I/O operations in some
+ <varname>io_context</varname>s and/or on some <varname>io_object</varname>s.
+ These rows are omitted from the view. For example, the checkpointer does not
+ checkpoint temporary tables, so there will be no rows for
+ <varname>backend_type</varname> <literal>checkpointer</literal> and
+ <varname>io_object</varname> <literal>temp relation</literal>.
+ </para>
+
+ <para>
+ In addition, some I/O operations will never be performed either by certain
+ <varname>backend_type</varname>s or in certain
+ <varname>io_context</varname>s or on certain <varname>io_object</varname>s.
+ These cells will be NULL. For example, temporary tables are not
+ <literal>fsync</literal>ed, so <varname>files_synced</varname> will be NULL
+ for <varname>io_object</varname> <literal>temp relation</literal>. Also, the
+ background writer does not perform reads, so <varname>read</varname> will be
+ NULL in rows for <varname>backend_type</varname> <literal>background
+ writer</literal>.
+ </para>
+
+ <para>
+ <structname>pg_stat_io</structname> can be used to inform database tuning.
+ For example:
+ <itemizedlist>
+ <listitem>
+ <para>
+ A high <varname>evicted</varname> count can indicate that shared buffers
+ should be increased.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Client backends rely on the checkpointer to ensure data is persisted to
+ permanent storage. Large numbers of <varname>files_synced</varname> by
+ <literal>client backend</literal>s could indicate a misconfiguration of
+ shared buffers or of the checkpointer. More information on configuring
+ checkpointer can be found in <xref linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Normally, client backends should be able to rely on auxiliary processes
+ like the checkpointer and the background writer to write out dirty data
+ as much as possible. Large numbers of writes by client backends could
+ indicate a misconfiguration of shared buffers or of the checkpointer.
+ More information on configuring the checkpointer can be found in <xref
+ linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
</sect2>
--
2.34.1
[text/x-patch] v46-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch (33.9K, ../../CAAKRu_bTjvHSd5tWWQkW8SWY-5frVp_vOJpFD+-V7jJG1Rf_Dw@mail.gmail.com/4-v46-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch)
download | inline diff:
From 57c9a605e59175ad4ac8857e40ff9cddc7909812 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 9 Jan 2023 14:42:25 -0500
Subject: [PATCH v46 4/5] Add system view tracking IO ops per backend type
Add pg_stat_io, a system view which tracks the number of IOOps
(evictions, reuses, reads, writes, extensions, and fsyncs) done on each
IOObject (relation, temp relation) in each IOContext ("normal" and those
using a BufferAccessStrategy) by each type of backend (e.g. client
backend, checkpointer).
Some BackendTypes do not accumulate IO operations statistics and will
not be included in the view.
Some IOContexts are not used by some BackendTypes and will not be in the
view. For example, checkpointer does not use a BufferAccessStrategy
(currently), so there will be no rows for BufferAccessStrategy
IOContexts for checkpointer.
Some IOObjects are never operated on in some IOContexts or by some
BackendTypes. These rows are omitted from the view. For example,
checkpointer will never operate on IOOBJECT_TEMP_RELATION data, so those
rows are omitted.
Some IOOps are invalid in combination with certain IOContexts and
certain IOObjects. Those cells will be NULL in the view to distinguish
between 0 observed IOOps of that type and an invalid combination. For
example, temporary tables are not fsynced so cells for all BackendTypes
for IOOBJECT_TEMP_RELATION and IOOP_FSYNC will be NULL.
Some BackendTypes never perform certain IOOps. Those cells will also be
NULL in the view. For example, bgwriter should not perform reads.
View stats are populated with statistics incremented when a backend
performs an IO Operation and maintained by the cumulative statistics
subsystem.
Each row of the view shows stats for a particular BackendType, IOObject,
IOContext combination (e.g. a client backend's operations on permanent
relations in shared buffers) and each column in the view is the total
number of IO Operations done (e.g. writes). So a cell in the view would
be, for example, the number of blocks of relation data written from
shared buffers by client backends since the last stats reset.
In anticipation of tracking WAL IO and non-block-oriented IO (such as
temporary file IO), the "op_bytes" column specifies the unit of the
"reads", "writes", and "extends" columns for a given row.
Note that some of the cells in the view are redundant with fields in
pg_stat_bgwriter (e.g. buffers_backend), however these have been kept in
pg_stat_bgwriter for backwards compatibility. Deriving the redundant
pg_stat_bgwriter stats from the IO operations stats structures was also
problematic due to the separate reset targets for 'bgwriter' and 'io'.
Suggested by Andres Freund
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
contrib/amcheck/expected/check_heap.out | 31 +++
contrib/amcheck/sql/check_heap.sql | 24 +++
src/backend/catalog/system_views.sql | 15 ++
src/backend/utils/adt/pgstatfuncs.c | 154 +++++++++++++++
src/include/catalog/pg_proc.dat | 9 +
src/test/regress/expected/rules.out | 12 ++
src/test/regress/expected/stats.out | 246 ++++++++++++++++++++++++
src/test/regress/sql/stats.sql | 150 +++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 642 insertions(+)
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index c010361025..2cf63302d5 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -66,6 +66,19 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy. This allows
+-- us to reliably test that pg_stat_io BULKREAD reads are being captured
+-- without relying on the size of shared buffers or on an expensive operation
+-- like CREATE DATABASE.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE test_stats LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE test_stats;
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -88,6 +101,23 @@ SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock :=
-------+--------+--------+-----
(0 rows)
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+ ?column?
+----------
+ t
+(1 row)
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
SET ROLE regress_heaptest_role;
@@ -195,6 +225,7 @@ ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE test_stats;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 298de6886a..a8e9c3c1e6 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -20,11 +20,26 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
+
-- Add some data so subsequent tests are not entirely trivial
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy. This allows
+-- us to reliably test that pg_stat_io BULKREAD reads are being captured
+-- without relying on the size of shared buffers or on an expensive operation
+-- like CREATE DATABASE.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE test_stats LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE test_stats;
+
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -32,6 +47,14 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');
SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
@@ -110,6 +133,7 @@ SELECT * FROM verify_heapam('test_foreign_table',
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE test_stats;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 447c9b970f..494a8791a5 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1117,6 +1117,21 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+CREATE VIEW pg_stat_io AS
+SELECT
+ b.backend_type,
+ b.io_context,
+ b.io_object,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+FROM pg_stat_get_io() b;
+
CREATE VIEW pg_stat_wal AS
SELECT
w.wal_records,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5c8e8336bf..51b941d4c8 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1234,6 +1234,160 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+* When adding a new column to the pg_stat_io view, add a new enum value
+* here above IO_NUM_COLUMNS.
+*/
+typedef enum io_stat_col
+{
+ IO_COL_BACKEND_TYPE,
+ IO_COL_IO_CONTEXT,
+ IO_COL_IO_OBJECT,
+ IO_COL_READS,
+ IO_COL_WRITES,
+ IO_COL_EXTENDS,
+ IO_COL_CONVERSION,
+ IO_COL_EVICTIONS,
+ IO_COL_REUSES,
+ IO_COL_FSYNCS,
+ IO_COL_RESET_TIME,
+ IO_NUM_COLUMNS,
+} io_stat_col;
+
+/*
+ * When adding a new IOOp, add a new io_stat_col and add a case to this
+ * function returning the corresponding io_stat_col.
+ */
+static io_stat_col
+pgstat_get_io_op_index(IOOp io_op)
+{
+ switch (io_op)
+ {
+ case IOOP_EVICT:
+ return IO_COL_EVICTIONS;
+ case IOOP_READ:
+ return IO_COL_READS;
+ case IOOP_REUSE:
+ return IO_COL_REUSES;
+ case IOOP_WRITE:
+ return IO_COL_WRITES;
+ case IOOP_EXTEND:
+ return IO_COL_EXTENDS;
+ case IOOP_FSYNC:
+ return IO_COL_FSYNCS;
+ }
+
+ elog(ERROR, "unrecognized IOOp value: %d", io_op);
+ pg_unreachable();
+}
+
+#ifdef USE_ASSERT_CHECKING
+static bool
+pgstat_iszero_io_object(const PgStat_Counter *obj)
+{
+ for (IOOp io_op = IOOP_EVICT; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ if (obj[io_op] != 0)
+ return false;
+ }
+
+ return true;
+}
+#endif
+
+Datum
+pg_stat_get_io(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo;
+ PgStat_IO *backends_io_stats;
+ Datum reset_time;
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ backends_io_stats = pgstat_fetch_stat_io();
+
+ reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
+
+ for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ {
+ bool bktype_tracked;
+ Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
+ PgStat_BackendIO *bktype_stats = &backends_io_stats->stats[bktype];
+
+ /*
+ * For those BackendTypes without IO Operation stats, skip
+ * representing them in the view altogether. We still loop through
+ * their counters so that we can assert that all values are zero.
+ */
+ bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOContext io_context = IOCONTEXT_BULKREAD;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ const char *context_name = pgstat_get_io_context_name(io_context);
+
+ for (IOObject io_obj = IOOBJECT_RELATION;
+ io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ {
+ const char *obj_name = pgstat_get_io_object_name(io_obj);
+
+ Datum values[IO_NUM_COLUMNS] = {0};
+ bool nulls[IO_NUM_COLUMNS] = {0};
+
+ /*
+ * Some combinations of IOContext, IOObject, and BackendType
+ * are not valid for any type of IOOp. In such cases, omit the
+ * entire row from the view.
+ */
+ if (!bktype_tracked ||
+ !pgstat_tracks_io_object(bktype, io_context, io_obj))
+ {
+ Assert(pgstat_iszero_io_object(bktype_stats->data[io_context][io_obj]));
+ continue;
+ }
+
+ values[IO_COL_BACKEND_TYPE] = bktype_desc;
+ values[IO_COL_IO_CONTEXT] = CStringGetTextDatum(context_name);
+ values[IO_COL_IO_OBJECT] = CStringGetTextDatum(obj_name);
+ values[IO_COL_RESET_TIME] = TimestampTzGetDatum(reset_time);
+
+ /*
+ * Hard-code this to the value of BLCKSZ for now. Future
+ * values could include XLOG_BLCKSZ, once WAL IO is tracked,
+ * and constant multipliers, once non-block-oriented IO (e.g.
+ * temporary file IO) is tracked.
+ */
+ values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
+
+ /*
+ * Some combinations of BackendType and IOOp, of IOContext and
+ * IOOp, and of IOObject and IOOp are not tracked. Set these
+ * cells in the view NULL and assert that these stats are zero
+ * as expected.
+ */
+ for (IOOp io_op = IOOP_EVICT; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ int col_idx = pgstat_get_io_op_index(io_op);
+
+ nulls[col_idx] = !pgstat_tracks_io_op(bktype, io_context, io_obj, io_op);
+
+ if (!nulls[col_idx])
+ values[col_idx] =
+ Int64GetDatum(bktype_stats->data[io_context][io_obj][io_op]);
+ else
+ Assert(bktype_stats->data[io_context][io_obj][io_op] == 0);
+ }
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ }
+ }
+
+ return (Datum) 0;
+}
+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3810de7b22..57a889cf49 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5690,6 +5690,15 @@
proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
+{ oid => '8459', descr => 'statistics: per backend type IO statistics',
+ proname => 'pg_stat_get_io', provolatile => 'v',
+ prorows => '30', proretset => 't',
+ proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,text,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{backend_type,io_context,io_object,reads,writes,extends,op_bytes,evictions,reuses,fsyncs,stats_reset}',
+ prosrc => 'pg_stat_get_io' },
+
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index fb9f936d43..6ae7882864 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1876,6 +1876,18 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_enc AS encrypted
FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
WHERE (s.client_port IS NOT NULL);
+pg_stat_io| SELECT b.backend_type,
+ b.io_context,
+ b.io_object,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+ FROM pg_stat_get_io() b(backend_type, io_context, io_object, reads, writes, extends, op_bytes, evictions, reuses, fsyncs, stats_reset);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 1d84407a03..a66fe86b05 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1126,4 +1126,250 @@ SELECT pg_stat_get_subscription_stats(NULL);
(1 row)
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_shared_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_extends_after > :io_sum_shared_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+-- The second checkpoint ensures that stats from the first checkpoint have been
+-- reported and protects against any potential races amongst the table
+-- creation, a possible timing-triggered checkpoint, and the explicit
+-- checkpoint in the test.
+SELECT sum(writes) AS io_sum_shared_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS io_sum_shared_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_writes_after > :io_sum_shared_writes_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT current_setting('fsync') = 'off' OR :io_sum_shared_fsyncs_after > :io_sum_shared_fsyncs_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+ count
+-------
+ 100
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS io_sum_shared_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_reads_after > :io_sum_shared_reads_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+-- Set temp_buffers to a low value so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO '1MB';
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS io_sum_local_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(evictions) AS io_sum_local_evictions_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 8000) as id, repeat('a', 100);
+SELECT sum(reads) AS io_sum_local_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+ count
+-------
+ 8000
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(evictions) AS io_sum_local_evictions_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(reads) AS io_sum_local_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(extends) AS io_sum_local_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_evictions_after > :io_sum_local_evictions_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_local_reads_after > :io_sum_local_reads_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_local_writes_after > :io_sum_local_writes_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_local_extends_after > :io_sum_local_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(writes) AS io_sum_local_writes_new_tblspc
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_writes_new_tblspc > :io_sum_local_writes_after;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT :io_sum_vac_strategy_reads_after > :io_sum_vac_strategy_reads_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_vac_strategy_reuses_after > :io_sum_vac_strategy_reuses_before;
+ ?column?
+----------
+ t
+(1 row)
+
+RESET wal_skip_threshold;
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test IO stats reset
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+ ?column?
+----------
+ t
+(1 row)
+
-- End of Stats Test
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index b4d6753c71..64b3da2765 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -536,4 +536,154 @@ SELECT pg_stat_get_replication_slot(NULL);
SELECT pg_stat_get_subscription_stats(NULL);
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_shared_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_extends_after > :io_sum_shared_extends_before;
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+-- The second checkpoint ensures that stats from the first checkpoint have been
+-- reported and protects against any potential races amongst the table
+-- creation, a possible timing-triggered checkpoint, and the explicit
+-- checkpoint in the test.
+SELECT sum(writes) AS io_sum_shared_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS io_sum_shared_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+
+SELECT :io_sum_shared_writes_after > :io_sum_shared_writes_before;
+SELECT current_setting('fsync') = 'off' OR :io_sum_shared_fsyncs_after > :io_sum_shared_fsyncs_before;
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS io_sum_shared_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_reads_after > :io_sum_shared_reads_before;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+
+-- Set temp_buffers to a low value so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO '1MB';
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS io_sum_local_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(evictions) AS io_sum_local_evictions_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 8000) as id, repeat('a', 100);
+
+SELECT sum(reads) AS io_sum_local_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+SELECT pg_stat_force_next_flush();
+SELECT sum(evictions) AS io_sum_local_evictions_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(reads) AS io_sum_local_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(extends) AS io_sum_local_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_evictions_after > :io_sum_local_evictions_before;
+SELECT :io_sum_local_reads_after > :io_sum_local_reads_before;
+SELECT :io_sum_local_writes_after > :io_sum_local_writes_before;
+SELECT :io_sum_local_extends_after > :io_sum_local_extends_before;
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+SELECT sum(writes) AS io_sum_local_writes_new_tblspc
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_writes_new_tblspc > :io_sum_local_writes_after;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT :io_sum_vac_strategy_reads_after > :io_sum_vac_strategy_reads_before;
+SELECT :io_sum_vac_strategy_reuses_after > :io_sum_vac_strategy_reuses_before;
+RESET wal_skip_threshold;
+
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+
+-- Test IO stats reset
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+
-- End of Stats Test
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7b66b1bc89..c4ecef2bf8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3377,6 +3377,7 @@ intset_internal_node
intset_leaf_node
intset_node
intvKEY
+io_stat_col
itemIdCompact
itemIdCompactData
iterator
--
2.34.1
[text/x-patch] v46-0003-pgstat-Count-IO-for-relations.patch (22.1K, ../../CAAKRu_bTjvHSd5tWWQkW8SWY-5frVp_vOJpFD+-V7jJG1Rf_Dw@mail.gmail.com/5-v46-0003-pgstat-Count-IO-for-relations.patch)
download | inline diff:
From 89ab692edfcf2af9e81fe6a45c3cad63158c8e20 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Wed, 4 Jan 2023 17:20:50 -0500
Subject: [PATCH v46 3/5] pgstat: Count IO for relations
Count IOOps done on IOObjects in IOContexts by various BackendTypes
using the IO stats infrastructure introduced by a previous commit.
The primary concern of these statistics is IO operations on data blocks
during the course of normal database operations. IO operations done by,
for example, the archiver or syslogger are not counted in these
statistics. WAL IO, temporary file IO, and IO done directly though smgr*
functions (such as when building an index) are not yet counted but would
be useful future additions.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
src/backend/storage/buffer/bufmgr.c | 109 ++++++++++++++++++++++----
src/backend/storage/buffer/freelist.c | 58 ++++++++++----
src/backend/storage/buffer/localbuf.c | 13 ++-
src/backend/storage/smgr/md.c | 25 ++++++
src/include/storage/buf_internals.h | 8 +-
src/include/storage/bufmgr.h | 7 +-
6 files changed, 184 insertions(+), 36 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8075828e8a..47d2b4c522 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -481,8 +481,9 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr);
-static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
+ bool *foundPtr, IOContext *io_context);
+static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
+ IOContext io_context, IOObject io_object);
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -823,6 +824,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BufferDesc *bufHdr;
Block bufBlock;
bool found;
+ IOContext io_context;
+ IOObject io_object;
bool isExtend;
bool isLocalBuf = SmgrIsTemp(smgr);
@@ -855,7 +858,14 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isLocalBuf)
{
- bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found);
+ /*
+ * LocalBufferAlloc() will set the io_context to IOCONTEXT_NORMAL. We
+ * do not use a BufferAccessStrategy for I/O of temporary tables.
+ * However, in some cases, the "strategy" may not be NULL, so we can't
+ * rely on IOContextForStrategy() to set the right IOContext for us.
+ * This may happen in cases like CREATE TEMPORARY TABLE AS...
+ */
+ bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found, &io_context);
if (found)
pgBufferUsage.local_blks_hit++;
else if (isExtend)
@@ -871,7 +881,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, &found, &io_context);
if (found)
pgBufferUsage.shared_blks_hit++;
else if (isExtend)
@@ -986,7 +996,16 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
Assert(!(pg_atomic_read_u32(&bufHdr->state) & BM_VALID)); /* spinlock not needed */
- bufBlock = isLocalBuf ? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr);
+ if (isLocalBuf)
+ {
+ bufBlock = LocalBufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ bufBlock = BufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_RELATION;
+ }
if (isExtend)
{
@@ -995,6 +1014,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
/* don't set checksum for all-zero page */
smgrextend(smgr, forkNum, blockNum, (char *) bufBlock, false);
+ pgstat_count_io_op(IOOP_EXTEND, io_object, io_context);
+
/*
* NB: we're *not* doing a ScheduleBufferTagForWriteback here;
* although we're essentially performing a write. At least on linux
@@ -1020,6 +1041,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgrread(smgr, forkNum, blockNum, (char *) bufBlock);
+ pgstat_count_io_op(IOOP_READ, io_object, io_context);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -1113,14 +1136,19 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* *foundPtr is actually redundant with the buffer's BM_VALID flag, but
* we keep it for simplicity in ReadBuffer.
*
+ * io_context is passed as an output parameter to avoid calling
+ * IOContextForStrategy() when there is a shared buffers hit and no IO
+ * statistics need be captured.
+ *
* No locks are held either at entry or exit.
*/
static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
+ bool from_ring;
BufferTag newTag; /* identity of requested block */
uint32 newHash; /* hash value for newTag */
LWLock *newPartitionLock; /* buffer partition lock for it */
@@ -1172,8 +1200,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* If we get here, previous attempts to read the buffer must
- * have failed ... but we shall bravely try again.
+ * have failed ... but we shall bravely try again. Set
+ * io_context since we will in fact need to count an IO
+ * Operation.
*/
+ *io_context = IOContextForStrategy(strategy);
*foundPtr = false;
}
}
@@ -1187,6 +1218,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ *io_context = IOContextForStrategy(strategy);
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1200,7 +1233,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* Select a victim buffer. The buffer is returned with its header
* spinlock still held!
*/
- buf = StrategyGetBuffer(strategy, &buf_state);
+ buf = StrategyGetBuffer(strategy, &buf_state, &from_ring);
Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 0);
@@ -1254,7 +1287,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
if (XLogNeedsFlush(lsn) &&
- StrategyRejectBuffer(strategy, buf))
+ StrategyRejectBuffer(strategy, buf, from_ring))
{
/* Drop lock/pin and loop around for another buffer */
LWLockRelease(BufferDescriptorGetContentLock(buf));
@@ -1269,7 +1302,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgr->smgr_rlocator.locator.dbOid,
smgr->smgr_rlocator.locator.relNumber);
- FlushBuffer(buf, NULL);
+ FlushBuffer(buf, NULL, *io_context, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(buf));
ScheduleBufferTagForWriteback(&BackendWritebackContext,
@@ -1441,6 +1474,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
+ if (oldFlags & BM_VALID)
+ {
+ /*
+ * When a BufferAccessStrategy is in use, blocks evicted from shared
+ * buffers are counted as IOOP_EVICT in the corresponding context
+ * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
+ * strategy in two cases: 1) while initially claiming buffers for the
+ * strategy ring 2) to replace an existing strategy ring buffer
+ * because it is pinned or in use and cannot be reused.
+ *
+ * Blocks evicted from buffers already in the strategy ring are
+ * counted as IOOP_REUSE in the corresponding strategy context.
+ *
+ * At this point, we can accurately count evictions and reuses,
+ * because we have successfully claimed the valid buffer. Previously,
+ * we may have been forced to release the buffer due to concurrent
+ * pinners or erroring out.
+ */
+ pgstat_count_io_op(from_ring ? IOOP_REUSE : IOOP_EVICT,
+ IOOBJECT_RELATION, *io_context);
+ }
+
if (oldPartitionLock != NULL)
{
BufTableDelete(&oldTag, oldHash);
@@ -2570,7 +2625,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
@@ -2820,7 +2875,7 @@ BufferGetTag(Buffer buffer, RelFileLocator *rlocator, ForkNumber *forknum,
* as the second parameter. If not, pass NULL.
*/
static void
-FlushBuffer(BufferDesc *buf, SMgrRelation reln)
+FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOContext io_context, IOObject io_object)
{
XLogRecPtr recptr;
ErrorContextCallback errcallback;
@@ -2912,6 +2967,26 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
bufToWrite,
false);
+ /*
+ * When a strategy is in use, only flushes of dirty buffers already in the
+ * strategy ring are counted as strategy writes (IOCONTEXT
+ * [BULKREAD|BULKWRITE|VACUUM] IOOP_WRITE) for the purpose of IO
+ * statistics tracking.
+ *
+ * If a shared buffer initially added to the ring must be flushed before
+ * being used, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE.
+ *
+ * If a shared buffer which was added to the ring later because the
+ * current strategy buffer is pinned or in use or because all strategy
+ * buffers were dirty and rejected (for BAS_BULKREAD operations only)
+ * requires flushing, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE
+ * (from_ring will be false).
+ *
+ * When a strategy is not in use, the write can only be a "regular" write
+ * of a dirty shared buffer (IOCONTEXT_NORMAL IOOP_WRITE).
+ */
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_RELATION, io_context);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -3554,6 +3629,8 @@ FlushRelationBuffers(Relation rel)
buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -3586,7 +3663,7 @@ FlushRelationBuffers(Relation rel)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, RelationGetSmgr(rel));
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3684,7 +3761,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, srelent->srel);
+ FlushBuffer(bufHdr, srelent->srel, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3894,7 +3971,7 @@ FlushDatabaseBuffers(Oid dbid)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3921,7 +3998,7 @@ FlushOneBuffer(Buffer buffer)
Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr)));
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7dec35801c..c690d5f15f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,7 @@
*/
#include "postgres.h"
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -81,12 +82,6 @@ typedef struct BufferAccessStrategyData
*/
int current;
- /*
- * True if the buffer just returned by StrategyGetBuffer had been in the
- * ring already.
- */
- bool current_was_in_ring;
-
/*
* Array of buffer numbers. InvalidBuffer (that is, zero) indicates we
* have not yet selected a buffer for this ring slot. For allocation
@@ -198,13 +193,15 @@ have_free_buffer(void)
* return the buffer with the buffer header spinlock still held.
*/
BufferDesc *
-StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
+StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring)
{
BufferDesc *buf;
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ *from_ring = false;
+
/*
* If given a strategy object, see whether it can select a buffer. We
* assume strategy objects don't need buffer_strategy_lock.
@@ -213,7 +210,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
{
buf = GetBufferFromRing(strategy, buf_state);
if (buf != NULL)
+ {
+ *from_ring = true;
return buf;
+ }
}
/*
@@ -602,7 +602,7 @@ FreeAccessStrategy(BufferAccessStrategy strategy)
/*
* GetBufferFromRing -- returns a buffer from the ring, or NULL if the
- * ring is empty.
+ * ring is empty / not usable.
*
* The bufhdr spin lock is held on the returned buffer.
*/
@@ -625,10 +625,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
*/
bufnum = strategy->buffers[strategy->current];
if (bufnum == InvalidBuffer)
- {
- strategy->current_was_in_ring = false;
return NULL;
- }
/*
* If the buffer is pinned we cannot use it under any circumstances.
@@ -644,7 +641,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) <= 1)
{
- strategy->current_was_in_ring = true;
*buf_state = local_buf_state;
return buf;
}
@@ -654,7 +650,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
* Tell caller to allocate a new buffer with the normal allocation
* strategy. He'll then replace this ring element via AddBufferToRing.
*/
- strategy->current_was_in_ring = false;
return NULL;
}
@@ -670,6 +665,39 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
strategy->buffers[strategy->current] = BufferDescriptorGetBuffer(buf);
}
+/*
+ * Utility function returning the IOContext of a given BufferAccessStrategy's
+ * strategy ring.
+ */
+IOContext
+IOContextForStrategy(BufferAccessStrategy strategy)
+{
+ if (!strategy)
+ return IOCONTEXT_NORMAL;
+
+ switch (strategy->btype)
+ {
+ case BAS_NORMAL:
+
+ /*
+ * Currently, GetAccessStrategy() returns NULL for
+ * BufferAccessStrategyType BAS_NORMAL, so this case is
+ * unreachable.
+ */
+ pg_unreachable();
+ return IOCONTEXT_NORMAL;
+ case BAS_BULKREAD:
+ return IOCONTEXT_BULKREAD;
+ case BAS_BULKWRITE:
+ return IOCONTEXT_BULKWRITE;
+ case BAS_VACUUM:
+ return IOCONTEXT_VACUUM;
+ }
+
+ elog(ERROR, "unrecognized BufferAccessStrategyType: %d", strategy->btype);
+ pg_unreachable();
+}
+
/*
* StrategyRejectBuffer -- consider rejecting a dirty buffer
*
@@ -682,14 +710,14 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
* if this buffer should be written and re-used.
*/
bool
-StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
+StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring)
{
/* We only do this in bulkread mode */
if (strategy->btype != BAS_BULKREAD)
return false;
/* Don't muck with behavior of normal buffer-replacement strategy */
- if (!strategy->current_was_in_ring ||
+ if (!from_ring ||
strategy->buffers[strategy->current] != BufferDescriptorGetBuffer(buf))
return false;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8372acc383..2108bbe7d8 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "catalog/catalog.h"
#include "executor/instrument.h"
+#include "pgstat.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "utils/guc_hooks.h"
@@ -107,7 +108,7 @@ PrefetchLocalBuffer(SMgrRelation smgr, ForkNumber forkNum,
*/
BufferDesc *
LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
BufferTag newTag; /* identity of requested block */
LocalBufferLookupEnt *hresult;
@@ -127,6 +128,14 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
+ /*
+ * IO Operations on local buffers are only done in IOCONTEXT_NORMAL. Set
+ * io_context here (instead of after a buffer hit would have returned) for
+ * convenience since we don't have to worry about the overhead of calling
+ * IOContextForStrategy().
+ */
+ *io_context = IOCONTEXT_NORMAL;
+
if (hresult)
{
b = hresult->id;
@@ -230,6 +239,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
buf_state &= ~BM_DIRTY;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
pgBufferUsage.local_blks_written++;
}
@@ -256,6 +266,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
ClearBufferTag(&bufHdr->tag);
buf_state &= ~(BM_VALID | BM_TAG_VALID);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_EVICT, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
}
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 60c9905eff..37bae4bf73 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -983,6 +983,15 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
{
MdfdVec *v = &reln->md_seg_fds[forknum][segno - 1];
+ /*
+ * fsyncs done through mdimmedsync() should be tracked in a separate
+ * IOContext than those done through mdsyncfiletag() to differentiate
+ * between unavoidable client backend fsyncs (e.g. those done during
+ * index build) and those which ideally would have been done by the
+ * checkpointer. Since other IO operations bypassing the buffer
+ * manager could also be tracked in such an IOContext, wait until
+ * these are also tracked to track immediate fsyncs.
+ */
if (FileSync(v->mdfd_vfd, WAIT_EVENT_DATA_FILE_IMMEDIATE_SYNC) < 0)
ereport(data_sync_elevel(ERROR),
(errcode_for_file_access(),
@@ -1021,6 +1030,19 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
{
+ /*
+ * We have no way of knowing if the current IOContext is
+ * IOCONTEXT_NORMAL or IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] at this
+ * point, so count the fsync as being in the IOCONTEXT_NORMAL
+ * IOContext. This is probably okay, because the number of backend
+ * fsyncs doesn't say anything about the efficacy of the
+ * BufferAccessStrategy. And counting both fsyncs done in
+ * IOCONTEXT_NORMAL and IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] under
+ * IOCONTEXT_NORMAL is likely clearer when investigating the number of
+ * backend fsyncs.
+ */
+ pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+
ereport(DEBUG1,
(errmsg_internal("could not forward fsync request because request queue is full")));
@@ -1410,6 +1432,9 @@ mdsyncfiletag(const FileTag *ftag, char *path)
if (need_to_close)
FileClose(file);
+ if (result >= 0)
+ pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+
errno = save_errno;
return result;
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index ed8aa2519c..0b44814740 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -15,6 +15,7 @@
#ifndef BUFMGR_INTERNALS_H
#define BUFMGR_INTERNALS_H
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
@@ -391,11 +392,12 @@ extern void IssuePendingWritebacks(WritebackContext *context);
extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *tag);
/* freelist.c */
+extern IOContext IOContextForStrategy(BufferAccessStrategy bas);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
- uint32 *buf_state);
+ uint32 *buf_state, bool *from_ring);
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
- BufferDesc *buf);
+ BufferDesc *buf, bool from_ring);
extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
extern void StrategyNotifyBgWriter(int bgwprocno);
@@ -417,7 +419,7 @@ extern PrefetchBufferResult PrefetchLocalBuffer(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum);
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
- BlockNumber blockNum, bool *foundPtr);
+ BlockNumber blockNum, bool *foundPtr, IOContext *io_context);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 33eadbc129..b8a18b8081 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -23,7 +23,12 @@
typedef void *Block;
-/* Possible arguments for GetAccessStrategy() */
+/*
+ * Possible arguments for GetAccessStrategy().
+ *
+ * If adding a new BufferAccessStrategyType, also add a new IOContext so
+ * IO statistics using this strategy are tracked.
+ */
typedef enum BufferAccessStrategyType
{
BAS_NORMAL, /* Normal random access */
--
2.34.1
[text/x-patch] v46-0002-pgstat-Infrastructure-to-track-IO-operations.patch (27.7K, ../../CAAKRu_bTjvHSd5tWWQkW8SWY-5frVp_vOJpFD+-V7jJG1Rf_Dw@mail.gmail.com/6-v46-0002-pgstat-Infrastructure-to-track-IO-operations.patch)
download | inline diff:
From 67a4cee981382eb63dd19c32458ef23da9eab5b9 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Wed, 4 Jan 2023 17:20:41 -0500
Subject: [PATCH v46 2/5] pgstat: Infrastructure to track IO operations
Introduce "IOOp", an IO operation done by a backend, "IOObject", the
target object of the IO, and "IOContext", the context or location of the
IO operations on that object. For example, the checkpointer may write a
shared buffer out. This would be considered an IOOP_WRITE IOOp on an
IOOBJECT_RELATION IOObject in the IOCONTEXT_NORMAL IOContext by
BackendType B_CHECKPOINTER.
Each IOOp (evict, extend, fsync, read, reuse, and write) can be counted
per IOObject (relation, temp relation) per IOContext (normal, bulkread,
bulkwrite, or vacuum) through a call to pgstat_count_io_op().
Note that this commit introduces the infrastructure to count IO
Operation statistics. A subsequent commit will add calls to
pgstat_count_io_op() in the appropriate locations.
IOContext IOCONTEXT_NORMAL concerns operations on local and shared
buffers, while IOCONTEXT_BULKREAD, IOCONTEXT_BULKWRITE, and
IOCONTEXT_VACUUM IOContexts concern IO operations on buffers as part of
a BufferAccessStrategy.
IOObject IOOBJECT_TEMP_RELATION concerns IO Operations on buffers
containing temporary table data, while IOObject IOOBJECT_RELATION
concerns IO Operations on buffers containing permanent relation data.
Stats on IOOps on all IOObjects in all IOContexts for a given backend
are first counted in a backend's local memory and then flushed to shared
memory and accumulated with those from all other backends, exited and
live.
Some BackendTypes will not flush their pending statistics at regular
intervals and explicitly call pgstat_flush_io_ops() during the course of
normal operations to flush their backend-local IO operation statistics
to shared memory in a timely manner.
Because not all BackendType, IOOp, IOObject, IOContext combinations are
valid, the validity of the stats is checked before flushing pending
stats and before reading in the existing stats file to shared memory.
The aggregated stats in shared memory could be extended in the future
with per-backend stats -- useful for per connection IO statistics and
monitoring.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Reviewed-by: Kyotaro Horiguchi <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 2 +
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 26 ++
src/backend/utils/activity/pgstat_bgwriter.c | 7 +-
.../utils/activity/pgstat_checkpointer.c | 7 +-
src/backend/utils/activity/pgstat_io.c | 377 ++++++++++++++++++
src/backend/utils/activity/pgstat_relation.c | 15 +-
src/backend/utils/activity/pgstat_shmem.c | 4 +
src/backend/utils/activity/pgstat_wal.c | 4 +-
src/backend/utils/adt/pgstatfuncs.c | 4 +-
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 66 +++
src/include/utils/pgstat_internal.h | 32 ++
src/tools/pgindent/typedefs.list | 6 +
15 files changed, 548 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_io.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 358d2ff90f..8d51ca3773 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5418,6 +5418,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
the <structname>pg_stat_bgwriter</structname>
view, <literal>archiver</literal> to reset all the counters shown in
the <structname>pg_stat_archiver</structname> view,
+ <literal>io</literal> to reset all the counters shown in the
+ <structname>pg_stat_io</structname> view,
<literal>wal</literal> to reset all the counters shown in the
<structname>pg_stat_wal</structname> view or
<literal>recovery_prefetch</literal> to reset all the counters shown
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index a80eda3cf4..7d7482dde0 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -22,6 +22,7 @@ OBJS = \
pgstat_checkpointer.o \
pgstat_database.o \
pgstat_function.o \
+ pgstat_io.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index a2b872c24b..518ee3f798 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -9,6 +9,7 @@ backend_sources += files(
'pgstat_checkpointer.c',
'pgstat_database.c',
'pgstat_function.c',
+ 'pgstat_io.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 0fa5370bcd..608c3b59da 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -72,6 +72,7 @@
* - pgstat_checkpointer.c
* - pgstat_database.c
* - pgstat_function.c
+ * - pgstat_io.c
* - pgstat_relation.c
* - pgstat_replslot.c
* - pgstat_slru.c
@@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
+ [PGSTAT_KIND_IO] = {
+ .name = "io_ops",
+
+ .fixed_amount = true,
+
+ .reset_all_cb = pgstat_io_reset_all_cb,
+ .snapshot_cb = pgstat_io_snapshot_cb,
+ },
+
[PGSTAT_KIND_SLRU] = {
.name = "slru",
@@ -582,6 +592,7 @@ pgstat_report_stat(bool force)
/* Don't expend a clock check if nothing to do */
if (dlist_is_empty(&pgStatPending) &&
+ !have_iostats &&
!have_slrustats &&
!pgstat_have_pending_wal())
{
@@ -628,6 +639,9 @@ pgstat_report_stat(bool force)
/* flush database / relation / function / ... stats */
partial_flush |= pgstat_flush_pending_entries(nowait);
+ /* flush IO stats */
+ partial_flush |= pgstat_flush_io(nowait);
+
/* flush wal stats */
partial_flush |= pgstat_flush_wal(nowait);
@@ -1322,6 +1336,12 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
+ /*
+ * Write IO stats struct
+ */
+ pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
+ write_chunk_s(fpout, &pgStatLocal.snapshot.io);
+
/*
* Write SLRU stats struct
*/
@@ -1496,6 +1516,12 @@ pgstat_read_statsfile(void)
if (!read_chunk_s(fpin, &shmem->checkpointer.stats))
goto error;
+ /*
+ * Read IO stats struct
+ */
+ if (!read_chunk_s(fpin, &shmem->io.stats))
+ goto error;
+
/*
* Read SLRU stats struct
*/
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 9247f2dda2..92be384b0d 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -24,7 +24,7 @@ PgStat_BgWriterStats PendingBgWriterStats = {0};
/*
- * Report bgwriter statistics
+ * Report bgwriter and IO statistics
*/
void
pgstat_report_bgwriter(void)
@@ -56,6 +56,11 @@ pgstat_report_bgwriter(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 3e9ab45103..26dec112f6 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -24,7 +24,7 @@ PgStat_CheckpointerStats PendingCheckpointerStats = {0};
/*
- * Report checkpointer statistics
+ * Report checkpointer and IO statistics
*/
void
pgstat_report_checkpointer(void)
@@ -62,6 +62,11 @@ pgstat_report_checkpointer(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingCheckpointerStats, 0, sizeof(PendingCheckpointerStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
new file mode 100644
index 0000000000..f95ab9c94d
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -0,0 +1,377 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_io.c
+ * Implementation of IO statistics.
+ *
+ * This file contains the implementation of IO statistics. It is kept separate
+ * from pgstat.c to enforce the line between the statistics access / storage
+ * implementation and the details about individual types of statistics.
+ *
+ * Copyright (c) 2021-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_io.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "utils/pgstat_internal.h"
+
+
+static PgStat_BackendIO PendingIOStats;
+bool have_iostats = false;
+
+
+void
+pgstat_count_io_op(IOOp io_op, IOObject io_object, IOContext io_context)
+{
+ Assert(io_context < IOCONTEXT_NUM_TYPES);
+ Assert(io_object < IOOBJECT_NUM_TYPES);
+ Assert(io_op < IOOP_NUM_TYPES);
+ Assert(pgstat_tracks_io_op(MyBackendType, io_context, io_object, io_op));
+
+ PendingIOStats.data[io_context][io_object][io_op]++;
+
+ have_iostats = true;
+}
+
+PgStat_IO *
+pgstat_fetch_stat_io(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_IO);
+
+ return &pgStatLocal.snapshot.io;
+}
+
+/*
+ * Flush out locally pending IO statistics
+ *
+ * If no stats have been recorded, this function returns false.
+ *
+ * If nowait is true, this function returns true if the lock could not be
+ * acquired. Otherwise, return false.
+ */
+bool
+pgstat_flush_io(bool nowait)
+{
+ LWLock *bktype_lock;
+ PgStat_BackendIO *bktype_shstats;
+
+ if (!have_iostats)
+ return false;
+
+ bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
+ bktype_shstats =
+ &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+
+ if (!nowait)
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
+ return true;
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOOp io_op = IOOP_FIRST;
+ io_op < IOOP_NUM_TYPES; io_op++)
+ bktype_shstats->data[io_context][io_object][io_op] +=
+ PendingIOStats.data[io_context][io_object][io_op];
+
+ Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
+
+ LWLockRelease(bktype_lock);
+
+ memset(&PendingIOStats, 0, sizeof(PendingIOStats));
+
+ have_iostats = false;
+
+ return false;
+}
+
+const char *
+pgstat_get_io_context_name(IOContext io_context)
+{
+ switch (io_context)
+ {
+ case IOCONTEXT_BULKREAD:
+ return "bulkread";
+ case IOCONTEXT_BULKWRITE:
+ return "bulkwrite";
+ case IOCONTEXT_NORMAL:
+ return "normal";
+ case IOCONTEXT_VACUUM:
+ return "vacuum";
+ }
+
+ elog(ERROR, "unrecognized IOContext value: %d", io_context);
+ pg_unreachable();
+}
+
+const char *
+pgstat_get_io_object_name(IOObject io_object)
+{
+ switch (io_object)
+ {
+ case IOOBJECT_RELATION:
+ return "relation";
+ case IOOBJECT_TEMP_RELATION:
+ return "temp relation";
+ }
+
+ elog(ERROR, "unrecognized IOObject value: %d", io_object);
+ pg_unreachable();
+}
+
+void
+pgstat_io_reset_all_cb(TimestampTz ts)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BackendIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BackendIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+
+ memset(bktype_shstats, 0, sizeof(*bktype_shstats));
+ LWLockRelease(bktype_lock);
+ }
+}
+
+void
+pgstat_io_snapshot_cb(void)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BackendIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStat_BackendIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_SHARED);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BackendIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.snapshot.io.stat_reset_timestamp =
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+
+ /* using struct assignment due to better type safety */
+ *bktype_snap = *bktype_shstats;
+ LWLockRelease(bktype_lock);
+ }
+}
+
+/*
+* IO statistics are not collected for all BackendTypes.
+*
+* The following BackendTypes do not participate in the cumulative stats
+* subsystem or do not perform IO on which we currently track:
+* - Syslogger because it is not connected to shared memory
+* - Archiver because most relevant archiving IO is delegated to a
+* specialized command or module
+* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
+*
+* Function returns true if BackendType participates in the cumulative stats
+* subsystem for IO and false if it does not.
+*/
+bool
+pgstat_tracks_io_bktype(BackendType bktype)
+{
+ /*
+ * List every type so that new backend types trigger a warning about
+ * needing to adjust this switch.
+ */
+ switch (bktype)
+ {
+ case B_INVALID:
+ case B_ARCHIVER:
+ case B_LOGGER:
+ case B_WAL_RECEIVER:
+ case B_WAL_WRITER:
+ return false;
+
+ case B_AUTOVAC_LAUNCHER:
+ case B_AUTOVAC_WORKER:
+ case B_BACKEND:
+ case B_BG_WORKER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_STANDALONE_BACKEND:
+ case B_STARTUP:
+ case B_WAL_SENDER:
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Some BackendTypes do not perform IO in certain IOContexts. Some IOObjects
+ * are never operated on in some IOContexts. Check that the given BackendType
+ * is expected to do IO in the given IOContext and that the given IOObject is
+ * expected to be operated on in the given IOContext.
+ */
+bool
+pgstat_tracks_io_object(BackendType bktype, IOContext io_context,
+ IOObject io_object)
+{
+ bool no_temp_rel;
+
+ /*
+ * Some BackendTypes should never track IO statistics.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ return false;
+
+ /*
+ * Currently, IO on temporary relations can only occur in the
+ * IOCONTEXT_NORMAL IOContext.
+ */
+ if (io_context != IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * In core Postgres, only regular backends and WAL Sender processes
+ * executing queries will use local buffers and operate on temporary
+ * relations. Parallel workers will not use local buffers (see
+ * InitLocalBuffers()); however, extensions leveraging background workers
+ * have no such limitation, so track IO on IOOBJECT_TEMP_RELATION for
+ * BackendType B_BG_WORKER.
+ */
+ no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
+ bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
+
+ if (no_temp_rel && io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * Some BackendTypes do not currently perform any IO in certain
+ * IOContexts, and, while it may not be inherently incorrect for them to
+ * do so, excluding those rows from the view makes the view easier to use.
+ */
+ if ((bktype == B_CHECKPOINTER || bktype == B_BG_WRITER) &&
+ (io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE ||
+ io_context == IOCONTEXT_VACUUM))
+ return false;
+
+ if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
+ return false;
+
+ if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
+ io_context == IOCONTEXT_BULKWRITE)
+ return false;
+
+ return true;
+}
+
+/*
+ * Some BackendTypes will never do certain IOOps and some IOOps should not
+ * occur in certain IOContexts. Check that the given IOOp is valid for the
+ * given BackendType in the given IOContext. Note that there are currently no
+ * cases of an IOOp being invalid for a particular BackendType only within a
+ * certain IOContext.
+ */
+bool
+pgstat_tracks_io_op(BackendType bktype, IOContext io_context,
+ IOObject io_object, IOOp io_op)
+{
+ bool strategy_io_context;
+
+ /* if (io_context, io_object) will never collect stats, we're done */
+ if (!pgstat_tracks_io_object(bktype, io_context, io_object))
+ return false;
+
+ /*
+ * Some BackendTypes will not do certain IOOps.
+ */
+ if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) &&
+ (io_op == IOOP_READ || io_op == IOOP_EVICT))
+ return false;
+
+ if ((bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER) && io_op == IOOP_EXTEND)
+ return false;
+
+ /*
+ * Some IOOps are not valid in certain IOContexts and some IOOps are only
+ * valid in certain contexts.
+ */
+ if (io_context == IOCONTEXT_BULKREAD && io_op == IOOP_EXTEND)
+ return false;
+
+ strategy_io_context = io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE || io_context == IOCONTEXT_VACUUM;
+
+ /*
+ * IOOP_REUSE is only relevant when a BufferAccessStrategy is in use.
+ */
+ if (!strategy_io_context && io_op == IOOP_REUSE)
+ return false;
+
+ /*
+ * IOOP_FSYNC IOOps done by a backend using a BufferAccessStrategy are
+ * counted in the IOCONTEXT_NORMAL IOContext. See comment in
+ * register_dirty_segment() for more details.
+ */
+ if (strategy_io_context && io_op == IOOP_FSYNC)
+ return false;
+
+ /*
+ * Temporary tables are not logged and thus do not require fsync'ing.
+ */
+ if (io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION && io_op == IOOP_FSYNC)
+ return false;
+
+ return true;
+}
+
+/*
+ * Check that stats have not been counted for any combination of IOContext,
+ * IOObject, and IOOp which are not tracked for the passed-in BackendType. The
+ * passed-in PgStat_BackendIO must contain stats from the BackendType specified
+ * by the second parameter. Caller is responsible for locking the passed-in
+ * PgStat_BackendIO, if needed.
+ */
+bool
+pgstat_bktype_io_stats_valid(PgStat_BackendIO *backend_io,
+ BackendType bktype)
+{
+ bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ /*
+ * Don't bother trying to skip to the next loop iteration if
+ * pgstat_tracks_io_object() would return false here. We still
+ * need to validate that each counter is zero anyway.
+ */
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ if ((!bktype_tracked || !pgstat_tracks_io_op(bktype, io_context, io_object, io_op)) &&
+ backend_io->data[io_context][io_object][io_op] != 0)
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 2e20b93c20..f793ac1516 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -206,7 +206,7 @@ pgstat_drop_relation(Relation rel)
}
/*
- * Report that the table was just vacuumed.
+ * Report that the table was just vacuumed and flush IO statistics.
*/
void
pgstat_report_vacuum(Oid tableoid, bool shared,
@@ -258,10 +258,18 @@ pgstat_report_vacuum(Oid tableoid, bool shared,
}
pgstat_unlock_entry(entry_ref);
+
+ /*
+ * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
+ * however this will not be called until after an entire autovacuum cycle
+ * is done -- which will likely vacuum many relations -- or until the
+ * VACUUM command has processed all tables and committed.
+ */
+ pgstat_flush_io(false);
}
/*
- * Report that the table was just analyzed.
+ * Report that the table was just analyzed and flush IO statistics.
*
* Caller must provide new live- and dead-tuples estimates, as well as a
* flag indicating whether to reset the mod_since_analyze counter.
@@ -341,6 +349,9 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
+
+ /* see pgstat_report_vacuum() */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index c1506b53d0..09fffd0e82 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -202,6 +202,10 @@ StatsShmemInit(void)
LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&ctl->io.locks[i],
+ LWTRANCHE_PGSTATS_DATA);
}
else
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index e7a82b5fed..e8598b2f4e 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -34,7 +34,7 @@ static WalUsage prevWalUsage;
/*
* Calculate how much WAL usage counters have increased and update
- * shared statistics.
+ * shared WAL and IO statistics.
*
* Must be called by processes that generate WAL, that do not call
* pgstat_report_stat(), like walwriter.
@@ -43,6 +43,8 @@ void
pgstat_report_wal(bool force)
{
pgstat_flush_wal(force);
+
+ pgstat_flush_io(force);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 58bd1360b9..5c8e8336bf 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1593,6 +1593,8 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_BGWRITER);
pgstat_reset_of_kind(PGSTAT_KIND_CHECKPOINTER);
}
+ else if (strcmp(target, "io") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_IO);
else if (strcmp(target, "recovery_prefetch") == 0)
XLogPrefetchResetStats();
else if (strcmp(target, "wal") == 0)
@@ -1601,7 +1603,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"io\", \"recovery_prefetch\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 96b3a1e1a0..c309e0233d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -332,6 +332,8 @@ typedef enum BackendType
B_WAL_WRITER,
} BackendType;
+#define BACKEND_NUM_TYPES (B_WAL_WRITER + 1)
+
extern PGDLLIMPORT BackendType MyBackendType;
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5e3326a3b9..f2a66ed7fb 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -48,6 +48,7 @@ typedef enum PgStat_Kind
PGSTAT_KIND_ARCHIVER,
PGSTAT_KIND_BGWRITER,
PGSTAT_KIND_CHECKPOINTER,
+ PGSTAT_KIND_IO,
PGSTAT_KIND_SLRU,
PGSTAT_KIND_WAL,
} PgStat_Kind;
@@ -276,6 +277,55 @@ typedef struct PgStat_CheckpointerStats
PgStat_Counter buf_fsync_backend;
} PgStat_CheckpointerStats;
+
+/*
+ * Types related to counting IO operations
+ */
+typedef enum IOContext
+{
+ IOCONTEXT_BULKREAD,
+ IOCONTEXT_BULKWRITE,
+ IOCONTEXT_NORMAL,
+ IOCONTEXT_VACUUM,
+} IOContext;
+
+#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
+#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
+
+typedef enum IOObject
+{
+ IOOBJECT_RELATION,
+ IOOBJECT_TEMP_RELATION,
+} IOObject;
+
+#define IOOBJECT_FIRST IOOBJECT_RELATION
+#define IOOBJECT_NUM_TYPES (IOOBJECT_TEMP_RELATION + 1)
+
+typedef enum IOOp
+{
+ IOOP_EVICT,
+ IOOP_EXTEND,
+ IOOP_FSYNC,
+ IOOP_READ,
+ IOOP_REUSE,
+ IOOP_WRITE,
+} IOOp;
+
+#define IOOP_FIRST IOOP_EVICT
+#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
+
+typedef struct PgStat_BackendIO
+{
+ PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
+} PgStat_BackendIO;
+
+typedef struct PgStat_IO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BackendIO stats[BACKEND_NUM_TYPES];
+} PgStat_IO;
+
+
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -453,6 +503,22 @@ extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern void pgstat_count_io_op(IOOp io_op, IOObject io_object, IOContext io_context);
+extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern const char *pgstat_get_io_context_name(IOContext io_context);
+extern const char *pgstat_get_io_object_name(IOObject io_object);
+
+extern bool pgstat_tracks_io_bktype(BackendType bktype);
+extern bool pgstat_tracks_io_object(BackendType bktype,
+ IOContext io_context, IOObject io_object);
+extern bool pgstat_tracks_io_op(BackendType bktype, IOContext io_context,
+ IOObject io_object, IOOp io_op);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 12fd51f1ae..bf8e4c3b8b 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -329,6 +329,17 @@ typedef struct PgStatShared_Checkpointer
PgStat_CheckpointerStats reset_offset;
} PgStatShared_Checkpointer;
+/* shared version of PgStat_IO */
+typedef struct PgStatShared_IO
+{
+ /*
+ * locks[i] protects stats.stats[i]. locks[0] also protects
+ * stats.stat_reset_timestamp.
+ */
+ LWLock locks[BACKEND_NUM_TYPES];
+ PgStat_IO stats;
+} PgStatShared_IO;
+
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -419,6 +430,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Archiver archiver;
PgStatShared_BgWriter bgwriter;
PgStatShared_Checkpointer checkpointer;
+ PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
} PgStat_ShmemControl;
@@ -442,6 +454,8 @@ typedef struct PgStat_Snapshot
PgStat_CheckpointerStats checkpointer;
+ PgStat_IO io;
+
PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS];
PgStat_WalStats wal;
@@ -549,6 +563,17 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern void pgstat_io_reset_all_cb(TimestampTz ts);
+extern void pgstat_io_snapshot_cb(void);
+extern bool pgstat_flush_io(bool nowait);
+extern bool pgstat_bktype_io_stats_valid(PgStat_BackendIO *context_ops,
+ BackendType bktype);
+
+
/*
* Functions in pgstat_relation.c
*/
@@ -643,6 +668,13 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid)
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
+/*
+ * Variables in pgstat_io.c
+ */
+
+extern PGDLLIMPORT bool have_iostats;
+
+
/*
* Variables in pgstat_slru.c
*/
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 23bafec5f7..7b66b1bc89 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1106,7 +1106,10 @@ ID
INFIX
INT128
INTERFACE_INFO
+IOContext
IOFuncSelector
+IOObject
+IOOp
IPCompareMethod
ITEM
IV
@@ -2016,6 +2019,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2033,6 +2037,8 @@ PgStat_FetchConsistency
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
+PgStat_IO
+PgStat_BackendIO
PgStat_Kind
PgStat_KindInfo
PgStat_LocalState
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-13 05:23 ` Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Justin Pryzby @ 2023-01-13 05:23 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Andres Freund <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; vignesh C <[email protected]>
On Thu, Jan 12, 2023 at 09:19:36PM -0500, Melanie Plageman wrote:
> On Wed, Jan 11, 2023 at 4:58 PM Justin Pryzby <[email protected]> wrote:
> >
> > > Subject: [PATCH v45 4/5] Add system view tracking IO ops per backend type
> >
> > The patch can/will fail with:
> >
> > CREATE TABLESPACE test_io_shared_stats_tblspc LOCATION '';
> > +WARNING: tablespaces created by regression test cases should have names starting with "regress_"
> >
> > CREATE TABLESPACE test_stats LOCATION '';
> > +WARNING: tablespaces created by regression test cases should have names starting with "regress_"
> >
> > (I already sent patches to address the omission in cirrus.yml)
>
> Thanks. I've fixed this
> I make a tablespace in amcheck -- are there recommendations for naming
> tablespaces in contrib also?
That's the test_stats one I mentioned.
Check with -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
> > > + <literal>bulkread</literal>: Qualifying large read I/O operations
> > > + done outside of shared buffers, for example, a sequential scan of a
> > > + large table.
> >
> > I don't think it's correct to say that it's "outside of" shared-buffers.
>
> I suppose "outside of" gives the wrong idea. But I need to make clear
> that this I/O is to and from buffers which are not a part of shared
> buffers right now -- they may still be accessible from the same data
> structures which access shared buffers but they are currently being used
> in a different way.
This would be a good place to link to a description of the ringbuffer,
if we had one.
> > s/Qualifying/Certain/
>
> I feel like qualifying is more specific than certain, but I would be open
> to changing it if there was a specific reason you don't like it.
I suggested to change it because at first I started to interpret it as
"The act of qualifying large I/O ops .." rather than "Large I/O ops that
qualify..".
+ Number of read operations of <varname>op_bytes</varname> size.
This is still a bit too easy to misinterpret as being in units of bytes.
I suggest: Number of read operations (which are each of the size
specified in >op_bytes<).
+ in order to add the shared buffer to a separate size-limited ring buffer
separate comma
+ More information on configuring checkpointer can be found in Section 30.5.
*the* checkpointer (as in the following paragraph)
+ <varname>backend_type</varname> <literal>checkpointer</literal> and
+ <varname>io_object</varname> <literal>temp relation</literal>.
+ </para>
I still think it's a bit hard to understand the <varname>s adjacent to
<literal>s.
+ Some backend_types
+ in some io_contexts
+ on some io_objects
+ in certain io_contexts
+ on certain io_objects
Maybe these should not use underscores: Some backend types never
perform I/O operations in some I/O contexts and/or on some i/o objects.
+ for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ for (IOContext io_context = IOCONTEXT_BULKREAD; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOObject io_obj = IOOBJECT_RELATION; io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ for (IOOp io_op = IOOP_EVICT; io_op < IOOP_NUM_TYPES; io_op++)
These look a bit fragile due to starting at some hardcoded "first"
value. In other places you use symbols "FIRST" symbols:
+ for (IOContext io_context = IOCONTEXT_FIRST; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOObject io_object = IOOBJECT_FIRST; io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
I think that's marginally better, but I think having to define both
FIRST and NUM is excessive and doesn't make it less fragile. Not sure
what anyone else will say, but I'd prefer if it started at "0".
Thanks for working on this - I'm looking forward to updating my rrdtool
script for this soon. It'll be nice to finally distinguish huge number
of "backend ringbuffer writes during ALTER" from other backend writes.
Currently, that makes it look like something is terribly wrong.
--
Justin
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
@ 2023-01-13 18:38 ` Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-16 21:41 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Maciek Sakrejda <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: Melanie Plageman @ 2023-01-13 18:38 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Andres Freund <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; vignesh C <[email protected]>
Attached is v47.
On Fri, Jan 13, 2023 at 12:23 AM Justin Pryzby <[email protected]> wrote:
>
> On Thu, Jan 12, 2023 at 09:19:36PM -0500, Melanie Plageman wrote:
> > On Wed, Jan 11, 2023 at 4:58 PM Justin Pryzby <[email protected]> wrote:
> > >
> > > > Subject: [PATCH v45 4/5] Add system view tracking IO ops per backend type
> > >
> > > The patch can/will fail with:
> > >
> > > CREATE TABLESPACE test_io_shared_stats_tblspc LOCATION '';
> > > +WARNING: tablespaces created by regression test cases should have names starting with "regress_"
> > >
> > > CREATE TABLESPACE test_stats LOCATION '';
> > > +WARNING: tablespaces created by regression test cases should have names starting with "regress_"
> > >
> > > (I already sent patches to address the omission in cirrus.yml)
> >
> > Thanks. I've fixed this
> > I make a tablespace in amcheck -- are there recommendations for naming
> > tablespaces in contrib also?
>
> That's the test_stats one I mentioned.
>
> Check with -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
Thanks. I have now changed both tablespace names and checked using that
macro.
> > > > + <literal>bulkread</literal>: Qualifying large read I/O operations
> > > > + done outside of shared buffers, for example, a sequential scan of a
> > > > + large table.
> > >
> > > I don't think it's correct to say that it's "outside of" shared-buffers.
> >
> > I suppose "outside of" gives the wrong idea. But I need to make clear
> > that this I/O is to and from buffers which are not a part of shared
> > buffers right now -- they may still be accessible from the same data
> > structures which access shared buffers but they are currently being used
> > in a different way.
>
> This would be a good place to link to a description of the ringbuffer,
> if we had one.
Indeed.
> > > s/Qualifying/Certain/
> >
> > I feel like qualifying is more specific than certain, but I would be open
> > to changing it if there was a specific reason you don't like it.
>
> I suggested to change it because at first I started to interpret it as
> "The act of qualifying large I/O ops .." rather than "Large I/O ops that
> qualify..".
I have changed it to "certain".
> + Number of read operations of <varname>op_bytes</varname> size.
>
> This is still a bit too easy to misinterpret as being in units of bytes.
> I suggest: Number of read operations (which are each of the size
> specified in >op_bytes<).
I have changed this.
> + in order to add the shared buffer to a separate size-limited ring buffer
>
> separate comma
>
> + More information on configuring checkpointer can be found in Section 30.5.
>
> *the* checkpointer (as in the following paragraph)
above items changed.
> + <varname>backend_type</varname> <literal>checkpointer</literal> and
> + <varname>io_object</varname> <literal>temp relation</literal>.
> + </para>
>
> I still think it's a bit hard to understand the <varname>s adjacent to
> <literal>s.
I agree it isn't great -- is there a different XML tag you suggest
instead of literal?
> + Some backend_types
> + in some io_contexts
> + on some io_objects
> + in certain io_contexts
> + on certain io_objects
>
> Maybe these should not use underscores: Some backend types never
> perform I/O operations in some I/O contexts and/or on some i/o objects.
I've changed this.
Also, taking another look, I forgot to update the docs' column name
tenses in the last version. That is now done.
> + for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
> + for (IOContext io_context = IOCONTEXT_BULKREAD; io_context < IOCONTEXT_NUM_TYPES; io_context++)
> + for (IOObject io_obj = IOOBJECT_RELATION; io_obj < IOOBJECT_NUM_TYPES; io_obj++)
> + for (IOOp io_op = IOOP_EVICT; io_op < IOOP_NUM_TYPES; io_op++)
>
> These look a bit fragile due to starting at some hardcoded "first"
> value. In other places you use symbols "FIRST" symbols:
>
> + for (IOContext io_context = IOCONTEXT_FIRST; io_context < IOCONTEXT_NUM_TYPES; io_context++)
> + for (IOObject io_object = IOOBJECT_FIRST; io_object < IOOBJECT_NUM_TYPES; io_object++)
> + for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
>
> I think that's marginally better, but I think having to define both
> FIRST and NUM is excessive and doesn't make it less fragile. Not sure
> what anyone else will say, but I'd prefer if it started at "0".
Thanks for catching the discrepancy in pg_stat_get_io(). I have changed
those instances to use _FIRST.
I think that having the loop start from the first enum value (except
when that value is something special like _INVALID like with
BackendType) is confusing. I agree that having multiple macros to allow
iteration through all enum values introduces some fragility. I'm not
sure about using the number 0 with the enum as the loop variable
data type. Is that a common pattern?
In this version, I have updated the loops in pg_stat_get_io() to use
_FIRST.
> Thanks for working on this - I'm looking forward to updating my rrdtool
> script for this soon. It'll be nice to finally distinguish huge number
> of "backend ringbuffer writes during ALTER" from other backend writes.
> Currently, that makes it look like something is terribly wrong.
Cool! I'm glad to know you will use it.
- Melanie
Attachments:
[text/x-patch] v47-0003-pgstat-Count-IO-for-relations.patch (22.1K, ../../CAAKRu_bHwGEbzNxxy+MQDkrsgog6aO6iUvajJ4d6PD98gFU7+w@mail.gmail.com/2-v47-0003-pgstat-Count-IO-for-relations.patch)
download | inline diff:
From ed7b2975732d599d8c809e21c2e33554c77ffc43 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Wed, 4 Jan 2023 17:20:50 -0500
Subject: [PATCH v47 3/5] pgstat: Count IO for relations
Count IOOps done on IOObjects in IOContexts by various BackendTypes
using the IO stats infrastructure introduced by a previous commit.
The primary concern of these statistics is IO operations on data blocks
during the course of normal database operations. IO operations done by,
for example, the archiver or syslogger are not counted in these
statistics. WAL IO, temporary file IO, and IO done directly though smgr*
functions (such as when building an index) are not yet counted but would
be useful future additions.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
src/backend/storage/buffer/bufmgr.c | 109 ++++++++++++++++++++++----
src/backend/storage/buffer/freelist.c | 58 ++++++++++----
src/backend/storage/buffer/localbuf.c | 13 ++-
src/backend/storage/smgr/md.c | 25 ++++++
src/include/storage/buf_internals.h | 8 +-
src/include/storage/bufmgr.h | 7 +-
6 files changed, 184 insertions(+), 36 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8075828e8a..47d2b4c522 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -481,8 +481,9 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr);
-static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
+ bool *foundPtr, IOContext *io_context);
+static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
+ IOContext io_context, IOObject io_object);
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -823,6 +824,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BufferDesc *bufHdr;
Block bufBlock;
bool found;
+ IOContext io_context;
+ IOObject io_object;
bool isExtend;
bool isLocalBuf = SmgrIsTemp(smgr);
@@ -855,7 +858,14 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isLocalBuf)
{
- bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found);
+ /*
+ * LocalBufferAlloc() will set the io_context to IOCONTEXT_NORMAL. We
+ * do not use a BufferAccessStrategy for I/O of temporary tables.
+ * However, in some cases, the "strategy" may not be NULL, so we can't
+ * rely on IOContextForStrategy() to set the right IOContext for us.
+ * This may happen in cases like CREATE TEMPORARY TABLE AS...
+ */
+ bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found, &io_context);
if (found)
pgBufferUsage.local_blks_hit++;
else if (isExtend)
@@ -871,7 +881,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, &found, &io_context);
if (found)
pgBufferUsage.shared_blks_hit++;
else if (isExtend)
@@ -986,7 +996,16 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
Assert(!(pg_atomic_read_u32(&bufHdr->state) & BM_VALID)); /* spinlock not needed */
- bufBlock = isLocalBuf ? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr);
+ if (isLocalBuf)
+ {
+ bufBlock = LocalBufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ bufBlock = BufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_RELATION;
+ }
if (isExtend)
{
@@ -995,6 +1014,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
/* don't set checksum for all-zero page */
smgrextend(smgr, forkNum, blockNum, (char *) bufBlock, false);
+ pgstat_count_io_op(IOOP_EXTEND, io_object, io_context);
+
/*
* NB: we're *not* doing a ScheduleBufferTagForWriteback here;
* although we're essentially performing a write. At least on linux
@@ -1020,6 +1041,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgrread(smgr, forkNum, blockNum, (char *) bufBlock);
+ pgstat_count_io_op(IOOP_READ, io_object, io_context);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -1113,14 +1136,19 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* *foundPtr is actually redundant with the buffer's BM_VALID flag, but
* we keep it for simplicity in ReadBuffer.
*
+ * io_context is passed as an output parameter to avoid calling
+ * IOContextForStrategy() when there is a shared buffers hit and no IO
+ * statistics need be captured.
+ *
* No locks are held either at entry or exit.
*/
static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
+ bool from_ring;
BufferTag newTag; /* identity of requested block */
uint32 newHash; /* hash value for newTag */
LWLock *newPartitionLock; /* buffer partition lock for it */
@@ -1172,8 +1200,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* If we get here, previous attempts to read the buffer must
- * have failed ... but we shall bravely try again.
+ * have failed ... but we shall bravely try again. Set
+ * io_context since we will in fact need to count an IO
+ * Operation.
*/
+ *io_context = IOContextForStrategy(strategy);
*foundPtr = false;
}
}
@@ -1187,6 +1218,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ *io_context = IOContextForStrategy(strategy);
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1200,7 +1233,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* Select a victim buffer. The buffer is returned with its header
* spinlock still held!
*/
- buf = StrategyGetBuffer(strategy, &buf_state);
+ buf = StrategyGetBuffer(strategy, &buf_state, &from_ring);
Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 0);
@@ -1254,7 +1287,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
if (XLogNeedsFlush(lsn) &&
- StrategyRejectBuffer(strategy, buf))
+ StrategyRejectBuffer(strategy, buf, from_ring))
{
/* Drop lock/pin and loop around for another buffer */
LWLockRelease(BufferDescriptorGetContentLock(buf));
@@ -1269,7 +1302,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgr->smgr_rlocator.locator.dbOid,
smgr->smgr_rlocator.locator.relNumber);
- FlushBuffer(buf, NULL);
+ FlushBuffer(buf, NULL, *io_context, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(buf));
ScheduleBufferTagForWriteback(&BackendWritebackContext,
@@ -1441,6 +1474,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
+ if (oldFlags & BM_VALID)
+ {
+ /*
+ * When a BufferAccessStrategy is in use, blocks evicted from shared
+ * buffers are counted as IOOP_EVICT in the corresponding context
+ * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
+ * strategy in two cases: 1) while initially claiming buffers for the
+ * strategy ring 2) to replace an existing strategy ring buffer
+ * because it is pinned or in use and cannot be reused.
+ *
+ * Blocks evicted from buffers already in the strategy ring are
+ * counted as IOOP_REUSE in the corresponding strategy context.
+ *
+ * At this point, we can accurately count evictions and reuses,
+ * because we have successfully claimed the valid buffer. Previously,
+ * we may have been forced to release the buffer due to concurrent
+ * pinners or erroring out.
+ */
+ pgstat_count_io_op(from_ring ? IOOP_REUSE : IOOP_EVICT,
+ IOOBJECT_RELATION, *io_context);
+ }
+
if (oldPartitionLock != NULL)
{
BufTableDelete(&oldTag, oldHash);
@@ -2570,7 +2625,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
@@ -2820,7 +2875,7 @@ BufferGetTag(Buffer buffer, RelFileLocator *rlocator, ForkNumber *forknum,
* as the second parameter. If not, pass NULL.
*/
static void
-FlushBuffer(BufferDesc *buf, SMgrRelation reln)
+FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOContext io_context, IOObject io_object)
{
XLogRecPtr recptr;
ErrorContextCallback errcallback;
@@ -2912,6 +2967,26 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
bufToWrite,
false);
+ /*
+ * When a strategy is in use, only flushes of dirty buffers already in the
+ * strategy ring are counted as strategy writes (IOCONTEXT
+ * [BULKREAD|BULKWRITE|VACUUM] IOOP_WRITE) for the purpose of IO
+ * statistics tracking.
+ *
+ * If a shared buffer initially added to the ring must be flushed before
+ * being used, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE.
+ *
+ * If a shared buffer which was added to the ring later because the
+ * current strategy buffer is pinned or in use or because all strategy
+ * buffers were dirty and rejected (for BAS_BULKREAD operations only)
+ * requires flushing, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE
+ * (from_ring will be false).
+ *
+ * When a strategy is not in use, the write can only be a "regular" write
+ * of a dirty shared buffer (IOCONTEXT_NORMAL IOOP_WRITE).
+ */
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_RELATION, io_context);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -3554,6 +3629,8 @@ FlushRelationBuffers(Relation rel)
buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -3586,7 +3663,7 @@ FlushRelationBuffers(Relation rel)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, RelationGetSmgr(rel));
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3684,7 +3761,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, srelent->srel);
+ FlushBuffer(bufHdr, srelent->srel, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3894,7 +3971,7 @@ FlushDatabaseBuffers(Oid dbid)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3921,7 +3998,7 @@ FlushOneBuffer(Buffer buffer)
Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr)));
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7dec35801c..c690d5f15f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,7 @@
*/
#include "postgres.h"
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -81,12 +82,6 @@ typedef struct BufferAccessStrategyData
*/
int current;
- /*
- * True if the buffer just returned by StrategyGetBuffer had been in the
- * ring already.
- */
- bool current_was_in_ring;
-
/*
* Array of buffer numbers. InvalidBuffer (that is, zero) indicates we
* have not yet selected a buffer for this ring slot. For allocation
@@ -198,13 +193,15 @@ have_free_buffer(void)
* return the buffer with the buffer header spinlock still held.
*/
BufferDesc *
-StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
+StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring)
{
BufferDesc *buf;
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ *from_ring = false;
+
/*
* If given a strategy object, see whether it can select a buffer. We
* assume strategy objects don't need buffer_strategy_lock.
@@ -213,7 +210,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
{
buf = GetBufferFromRing(strategy, buf_state);
if (buf != NULL)
+ {
+ *from_ring = true;
return buf;
+ }
}
/*
@@ -602,7 +602,7 @@ FreeAccessStrategy(BufferAccessStrategy strategy)
/*
* GetBufferFromRing -- returns a buffer from the ring, or NULL if the
- * ring is empty.
+ * ring is empty / not usable.
*
* The bufhdr spin lock is held on the returned buffer.
*/
@@ -625,10 +625,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
*/
bufnum = strategy->buffers[strategy->current];
if (bufnum == InvalidBuffer)
- {
- strategy->current_was_in_ring = false;
return NULL;
- }
/*
* If the buffer is pinned we cannot use it under any circumstances.
@@ -644,7 +641,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) <= 1)
{
- strategy->current_was_in_ring = true;
*buf_state = local_buf_state;
return buf;
}
@@ -654,7 +650,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
* Tell caller to allocate a new buffer with the normal allocation
* strategy. He'll then replace this ring element via AddBufferToRing.
*/
- strategy->current_was_in_ring = false;
return NULL;
}
@@ -670,6 +665,39 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
strategy->buffers[strategy->current] = BufferDescriptorGetBuffer(buf);
}
+/*
+ * Utility function returning the IOContext of a given BufferAccessStrategy's
+ * strategy ring.
+ */
+IOContext
+IOContextForStrategy(BufferAccessStrategy strategy)
+{
+ if (!strategy)
+ return IOCONTEXT_NORMAL;
+
+ switch (strategy->btype)
+ {
+ case BAS_NORMAL:
+
+ /*
+ * Currently, GetAccessStrategy() returns NULL for
+ * BufferAccessStrategyType BAS_NORMAL, so this case is
+ * unreachable.
+ */
+ pg_unreachable();
+ return IOCONTEXT_NORMAL;
+ case BAS_BULKREAD:
+ return IOCONTEXT_BULKREAD;
+ case BAS_BULKWRITE:
+ return IOCONTEXT_BULKWRITE;
+ case BAS_VACUUM:
+ return IOCONTEXT_VACUUM;
+ }
+
+ elog(ERROR, "unrecognized BufferAccessStrategyType: %d", strategy->btype);
+ pg_unreachable();
+}
+
/*
* StrategyRejectBuffer -- consider rejecting a dirty buffer
*
@@ -682,14 +710,14 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
* if this buffer should be written and re-used.
*/
bool
-StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
+StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring)
{
/* We only do this in bulkread mode */
if (strategy->btype != BAS_BULKREAD)
return false;
/* Don't muck with behavior of normal buffer-replacement strategy */
- if (!strategy->current_was_in_ring ||
+ if (!from_ring ||
strategy->buffers[strategy->current] != BufferDescriptorGetBuffer(buf))
return false;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8372acc383..2108bbe7d8 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "catalog/catalog.h"
#include "executor/instrument.h"
+#include "pgstat.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "utils/guc_hooks.h"
@@ -107,7 +108,7 @@ PrefetchLocalBuffer(SMgrRelation smgr, ForkNumber forkNum,
*/
BufferDesc *
LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
BufferTag newTag; /* identity of requested block */
LocalBufferLookupEnt *hresult;
@@ -127,6 +128,14 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
+ /*
+ * IO Operations on local buffers are only done in IOCONTEXT_NORMAL. Set
+ * io_context here (instead of after a buffer hit would have returned) for
+ * convenience since we don't have to worry about the overhead of calling
+ * IOContextForStrategy().
+ */
+ *io_context = IOCONTEXT_NORMAL;
+
if (hresult)
{
b = hresult->id;
@@ -230,6 +239,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
buf_state &= ~BM_DIRTY;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
pgBufferUsage.local_blks_written++;
}
@@ -256,6 +266,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
ClearBufferTag(&bufHdr->tag);
buf_state &= ~(BM_VALID | BM_TAG_VALID);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_EVICT, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
}
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 60c9905eff..37bae4bf73 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -983,6 +983,15 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
{
MdfdVec *v = &reln->md_seg_fds[forknum][segno - 1];
+ /*
+ * fsyncs done through mdimmedsync() should be tracked in a separate
+ * IOContext than those done through mdsyncfiletag() to differentiate
+ * between unavoidable client backend fsyncs (e.g. those done during
+ * index build) and those which ideally would have been done by the
+ * checkpointer. Since other IO operations bypassing the buffer
+ * manager could also be tracked in such an IOContext, wait until
+ * these are also tracked to track immediate fsyncs.
+ */
if (FileSync(v->mdfd_vfd, WAIT_EVENT_DATA_FILE_IMMEDIATE_SYNC) < 0)
ereport(data_sync_elevel(ERROR),
(errcode_for_file_access(),
@@ -1021,6 +1030,19 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
{
+ /*
+ * We have no way of knowing if the current IOContext is
+ * IOCONTEXT_NORMAL or IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] at this
+ * point, so count the fsync as being in the IOCONTEXT_NORMAL
+ * IOContext. This is probably okay, because the number of backend
+ * fsyncs doesn't say anything about the efficacy of the
+ * BufferAccessStrategy. And counting both fsyncs done in
+ * IOCONTEXT_NORMAL and IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] under
+ * IOCONTEXT_NORMAL is likely clearer when investigating the number of
+ * backend fsyncs.
+ */
+ pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+
ereport(DEBUG1,
(errmsg_internal("could not forward fsync request because request queue is full")));
@@ -1410,6 +1432,9 @@ mdsyncfiletag(const FileTag *ftag, char *path)
if (need_to_close)
FileClose(file);
+ if (result >= 0)
+ pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+
errno = save_errno;
return result;
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index ed8aa2519c..0b44814740 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -15,6 +15,7 @@
#ifndef BUFMGR_INTERNALS_H
#define BUFMGR_INTERNALS_H
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
@@ -391,11 +392,12 @@ extern void IssuePendingWritebacks(WritebackContext *context);
extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *tag);
/* freelist.c */
+extern IOContext IOContextForStrategy(BufferAccessStrategy bas);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
- uint32 *buf_state);
+ uint32 *buf_state, bool *from_ring);
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
- BufferDesc *buf);
+ BufferDesc *buf, bool from_ring);
extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
extern void StrategyNotifyBgWriter(int bgwprocno);
@@ -417,7 +419,7 @@ extern PrefetchBufferResult PrefetchLocalBuffer(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum);
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
- BlockNumber blockNum, bool *foundPtr);
+ BlockNumber blockNum, bool *foundPtr, IOContext *io_context);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 33eadbc129..b8a18b8081 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -23,7 +23,12 @@
typedef void *Block;
-/* Possible arguments for GetAccessStrategy() */
+/*
+ * Possible arguments for GetAccessStrategy().
+ *
+ * If adding a new BufferAccessStrategyType, also add a new IOContext so
+ * IO statistics using this strategy are tracked.
+ */
typedef enum BufferAccessStrategyType
{
BAS_NORMAL, /* Normal random access */
--
2.34.1
[text/x-patch] v47-0002-pgstat-Infrastructure-to-track-IO-operations.patch (27.7K, ../../CAAKRu_bHwGEbzNxxy+MQDkrsgog6aO6iUvajJ4d6PD98gFU7+w@mail.gmail.com/3-v47-0002-pgstat-Infrastructure-to-track-IO-operations.patch)
download | inline diff:
From 28a57a119e2d74e00b329da850c75584178e6967 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Wed, 4 Jan 2023 17:20:41 -0500
Subject: [PATCH v47 2/5] pgstat: Infrastructure to track IO operations
Introduce "IOOp", an IO operation done by a backend, "IOObject", the
target object of the IO, and "IOContext", the context or location of the
IO operations on that object. For example, the checkpointer may write a
shared buffer out. This would be considered an IOOP_WRITE IOOp on an
IOOBJECT_RELATION IOObject in the IOCONTEXT_NORMAL IOContext by
BackendType B_CHECKPOINTER.
Each IOOp (evict, extend, fsync, read, reuse, and write) can be counted
per IOObject (relation, temp relation) per IOContext (normal, bulkread,
bulkwrite, or vacuum) through a call to pgstat_count_io_op().
Note that this commit introduces the infrastructure to count IO
Operation statistics. A subsequent commit will add calls to
pgstat_count_io_op() in the appropriate locations.
IOContext IOCONTEXT_NORMAL concerns operations on local and shared
buffers, while IOCONTEXT_BULKREAD, IOCONTEXT_BULKWRITE, and
IOCONTEXT_VACUUM IOContexts concern IO operations on buffers as part of
a BufferAccessStrategy.
IOObject IOOBJECT_TEMP_RELATION concerns IO Operations on buffers
containing temporary table data, while IOObject IOOBJECT_RELATION
concerns IO Operations on buffers containing permanent relation data.
Stats on IOOps on all IOObjects in all IOContexts for a given backend
are first counted in a backend's local memory and then flushed to shared
memory and accumulated with those from all other backends, exited and
live.
Some BackendTypes will not flush their pending statistics at regular
intervals and explicitly call pgstat_flush_io_ops() during the course of
normal operations to flush their backend-local IO operation statistics
to shared memory in a timely manner.
Because not all BackendType, IOOp, IOObject, IOContext combinations are
valid, the validity of the stats is checked before flushing pending
stats and before reading in the existing stats file to shared memory.
The aggregated stats in shared memory could be extended in the future
with per-backend stats -- useful for per connection IO statistics and
monitoring.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Reviewed-by: Kyotaro Horiguchi <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 2 +
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 26 ++
src/backend/utils/activity/pgstat_bgwriter.c | 7 +-
.../utils/activity/pgstat_checkpointer.c | 7 +-
src/backend/utils/activity/pgstat_io.c | 377 ++++++++++++++++++
src/backend/utils/activity/pgstat_relation.c | 15 +-
src/backend/utils/activity/pgstat_shmem.c | 4 +
src/backend/utils/activity/pgstat_wal.c | 4 +-
src/backend/utils/adt/pgstatfuncs.c | 4 +-
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 66 +++
src/include/utils/pgstat_internal.h | 32 ++
src/tools/pgindent/typedefs.list | 6 +
15 files changed, 548 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_io.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 358d2ff90f..8d51ca3773 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5418,6 +5418,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
the <structname>pg_stat_bgwriter</structname>
view, <literal>archiver</literal> to reset all the counters shown in
the <structname>pg_stat_archiver</structname> view,
+ <literal>io</literal> to reset all the counters shown in the
+ <structname>pg_stat_io</structname> view,
<literal>wal</literal> to reset all the counters shown in the
<structname>pg_stat_wal</structname> view or
<literal>recovery_prefetch</literal> to reset all the counters shown
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index a80eda3cf4..7d7482dde0 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -22,6 +22,7 @@ OBJS = \
pgstat_checkpointer.o \
pgstat_database.o \
pgstat_function.o \
+ pgstat_io.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index a2b872c24b..518ee3f798 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -9,6 +9,7 @@ backend_sources += files(
'pgstat_checkpointer.c',
'pgstat_database.c',
'pgstat_function.c',
+ 'pgstat_io.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 0fa5370bcd..608c3b59da 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -72,6 +72,7 @@
* - pgstat_checkpointer.c
* - pgstat_database.c
* - pgstat_function.c
+ * - pgstat_io.c
* - pgstat_relation.c
* - pgstat_replslot.c
* - pgstat_slru.c
@@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
+ [PGSTAT_KIND_IO] = {
+ .name = "io_ops",
+
+ .fixed_amount = true,
+
+ .reset_all_cb = pgstat_io_reset_all_cb,
+ .snapshot_cb = pgstat_io_snapshot_cb,
+ },
+
[PGSTAT_KIND_SLRU] = {
.name = "slru",
@@ -582,6 +592,7 @@ pgstat_report_stat(bool force)
/* Don't expend a clock check if nothing to do */
if (dlist_is_empty(&pgStatPending) &&
+ !have_iostats &&
!have_slrustats &&
!pgstat_have_pending_wal())
{
@@ -628,6 +639,9 @@ pgstat_report_stat(bool force)
/* flush database / relation / function / ... stats */
partial_flush |= pgstat_flush_pending_entries(nowait);
+ /* flush IO stats */
+ partial_flush |= pgstat_flush_io(nowait);
+
/* flush wal stats */
partial_flush |= pgstat_flush_wal(nowait);
@@ -1322,6 +1336,12 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
+ /*
+ * Write IO stats struct
+ */
+ pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
+ write_chunk_s(fpout, &pgStatLocal.snapshot.io);
+
/*
* Write SLRU stats struct
*/
@@ -1496,6 +1516,12 @@ pgstat_read_statsfile(void)
if (!read_chunk_s(fpin, &shmem->checkpointer.stats))
goto error;
+ /*
+ * Read IO stats struct
+ */
+ if (!read_chunk_s(fpin, &shmem->io.stats))
+ goto error;
+
/*
* Read SLRU stats struct
*/
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 9247f2dda2..92be384b0d 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -24,7 +24,7 @@ PgStat_BgWriterStats PendingBgWriterStats = {0};
/*
- * Report bgwriter statistics
+ * Report bgwriter and IO statistics
*/
void
pgstat_report_bgwriter(void)
@@ -56,6 +56,11 @@ pgstat_report_bgwriter(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 3e9ab45103..26dec112f6 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -24,7 +24,7 @@ PgStat_CheckpointerStats PendingCheckpointerStats = {0};
/*
- * Report checkpointer statistics
+ * Report checkpointer and IO statistics
*/
void
pgstat_report_checkpointer(void)
@@ -62,6 +62,11 @@ pgstat_report_checkpointer(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingCheckpointerStats, 0, sizeof(PendingCheckpointerStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
new file mode 100644
index 0000000000..f95ab9c94d
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -0,0 +1,377 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_io.c
+ * Implementation of IO statistics.
+ *
+ * This file contains the implementation of IO statistics. It is kept separate
+ * from pgstat.c to enforce the line between the statistics access / storage
+ * implementation and the details about individual types of statistics.
+ *
+ * Copyright (c) 2021-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_io.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "utils/pgstat_internal.h"
+
+
+static PgStat_BackendIO PendingIOStats;
+bool have_iostats = false;
+
+
+void
+pgstat_count_io_op(IOOp io_op, IOObject io_object, IOContext io_context)
+{
+ Assert(io_context < IOCONTEXT_NUM_TYPES);
+ Assert(io_object < IOOBJECT_NUM_TYPES);
+ Assert(io_op < IOOP_NUM_TYPES);
+ Assert(pgstat_tracks_io_op(MyBackendType, io_context, io_object, io_op));
+
+ PendingIOStats.data[io_context][io_object][io_op]++;
+
+ have_iostats = true;
+}
+
+PgStat_IO *
+pgstat_fetch_stat_io(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_IO);
+
+ return &pgStatLocal.snapshot.io;
+}
+
+/*
+ * Flush out locally pending IO statistics
+ *
+ * If no stats have been recorded, this function returns false.
+ *
+ * If nowait is true, this function returns true if the lock could not be
+ * acquired. Otherwise, return false.
+ */
+bool
+pgstat_flush_io(bool nowait)
+{
+ LWLock *bktype_lock;
+ PgStat_BackendIO *bktype_shstats;
+
+ if (!have_iostats)
+ return false;
+
+ bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
+ bktype_shstats =
+ &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+
+ if (!nowait)
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
+ return true;
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOOp io_op = IOOP_FIRST;
+ io_op < IOOP_NUM_TYPES; io_op++)
+ bktype_shstats->data[io_context][io_object][io_op] +=
+ PendingIOStats.data[io_context][io_object][io_op];
+
+ Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
+
+ LWLockRelease(bktype_lock);
+
+ memset(&PendingIOStats, 0, sizeof(PendingIOStats));
+
+ have_iostats = false;
+
+ return false;
+}
+
+const char *
+pgstat_get_io_context_name(IOContext io_context)
+{
+ switch (io_context)
+ {
+ case IOCONTEXT_BULKREAD:
+ return "bulkread";
+ case IOCONTEXT_BULKWRITE:
+ return "bulkwrite";
+ case IOCONTEXT_NORMAL:
+ return "normal";
+ case IOCONTEXT_VACUUM:
+ return "vacuum";
+ }
+
+ elog(ERROR, "unrecognized IOContext value: %d", io_context);
+ pg_unreachable();
+}
+
+const char *
+pgstat_get_io_object_name(IOObject io_object)
+{
+ switch (io_object)
+ {
+ case IOOBJECT_RELATION:
+ return "relation";
+ case IOOBJECT_TEMP_RELATION:
+ return "temp relation";
+ }
+
+ elog(ERROR, "unrecognized IOObject value: %d", io_object);
+ pg_unreachable();
+}
+
+void
+pgstat_io_reset_all_cb(TimestampTz ts)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BackendIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BackendIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+
+ memset(bktype_shstats, 0, sizeof(*bktype_shstats));
+ LWLockRelease(bktype_lock);
+ }
+}
+
+void
+pgstat_io_snapshot_cb(void)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BackendIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStat_BackendIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_SHARED);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BackendIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.snapshot.io.stat_reset_timestamp =
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+
+ /* using struct assignment due to better type safety */
+ *bktype_snap = *bktype_shstats;
+ LWLockRelease(bktype_lock);
+ }
+}
+
+/*
+* IO statistics are not collected for all BackendTypes.
+*
+* The following BackendTypes do not participate in the cumulative stats
+* subsystem or do not perform IO on which we currently track:
+* - Syslogger because it is not connected to shared memory
+* - Archiver because most relevant archiving IO is delegated to a
+* specialized command or module
+* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
+*
+* Function returns true if BackendType participates in the cumulative stats
+* subsystem for IO and false if it does not.
+*/
+bool
+pgstat_tracks_io_bktype(BackendType bktype)
+{
+ /*
+ * List every type so that new backend types trigger a warning about
+ * needing to adjust this switch.
+ */
+ switch (bktype)
+ {
+ case B_INVALID:
+ case B_ARCHIVER:
+ case B_LOGGER:
+ case B_WAL_RECEIVER:
+ case B_WAL_WRITER:
+ return false;
+
+ case B_AUTOVAC_LAUNCHER:
+ case B_AUTOVAC_WORKER:
+ case B_BACKEND:
+ case B_BG_WORKER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_STANDALONE_BACKEND:
+ case B_STARTUP:
+ case B_WAL_SENDER:
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Some BackendTypes do not perform IO in certain IOContexts. Some IOObjects
+ * are never operated on in some IOContexts. Check that the given BackendType
+ * is expected to do IO in the given IOContext and that the given IOObject is
+ * expected to be operated on in the given IOContext.
+ */
+bool
+pgstat_tracks_io_object(BackendType bktype, IOContext io_context,
+ IOObject io_object)
+{
+ bool no_temp_rel;
+
+ /*
+ * Some BackendTypes should never track IO statistics.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ return false;
+
+ /*
+ * Currently, IO on temporary relations can only occur in the
+ * IOCONTEXT_NORMAL IOContext.
+ */
+ if (io_context != IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * In core Postgres, only regular backends and WAL Sender processes
+ * executing queries will use local buffers and operate on temporary
+ * relations. Parallel workers will not use local buffers (see
+ * InitLocalBuffers()); however, extensions leveraging background workers
+ * have no such limitation, so track IO on IOOBJECT_TEMP_RELATION for
+ * BackendType B_BG_WORKER.
+ */
+ no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
+ bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
+
+ if (no_temp_rel && io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * Some BackendTypes do not currently perform any IO in certain
+ * IOContexts, and, while it may not be inherently incorrect for them to
+ * do so, excluding those rows from the view makes the view easier to use.
+ */
+ if ((bktype == B_CHECKPOINTER || bktype == B_BG_WRITER) &&
+ (io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE ||
+ io_context == IOCONTEXT_VACUUM))
+ return false;
+
+ if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
+ return false;
+
+ if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
+ io_context == IOCONTEXT_BULKWRITE)
+ return false;
+
+ return true;
+}
+
+/*
+ * Some BackendTypes will never do certain IOOps and some IOOps should not
+ * occur in certain IOContexts. Check that the given IOOp is valid for the
+ * given BackendType in the given IOContext. Note that there are currently no
+ * cases of an IOOp being invalid for a particular BackendType only within a
+ * certain IOContext.
+ */
+bool
+pgstat_tracks_io_op(BackendType bktype, IOContext io_context,
+ IOObject io_object, IOOp io_op)
+{
+ bool strategy_io_context;
+
+ /* if (io_context, io_object) will never collect stats, we're done */
+ if (!pgstat_tracks_io_object(bktype, io_context, io_object))
+ return false;
+
+ /*
+ * Some BackendTypes will not do certain IOOps.
+ */
+ if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) &&
+ (io_op == IOOP_READ || io_op == IOOP_EVICT))
+ return false;
+
+ if ((bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER) && io_op == IOOP_EXTEND)
+ return false;
+
+ /*
+ * Some IOOps are not valid in certain IOContexts and some IOOps are only
+ * valid in certain contexts.
+ */
+ if (io_context == IOCONTEXT_BULKREAD && io_op == IOOP_EXTEND)
+ return false;
+
+ strategy_io_context = io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE || io_context == IOCONTEXT_VACUUM;
+
+ /*
+ * IOOP_REUSE is only relevant when a BufferAccessStrategy is in use.
+ */
+ if (!strategy_io_context && io_op == IOOP_REUSE)
+ return false;
+
+ /*
+ * IOOP_FSYNC IOOps done by a backend using a BufferAccessStrategy are
+ * counted in the IOCONTEXT_NORMAL IOContext. See comment in
+ * register_dirty_segment() for more details.
+ */
+ if (strategy_io_context && io_op == IOOP_FSYNC)
+ return false;
+
+ /*
+ * Temporary tables are not logged and thus do not require fsync'ing.
+ */
+ if (io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION && io_op == IOOP_FSYNC)
+ return false;
+
+ return true;
+}
+
+/*
+ * Check that stats have not been counted for any combination of IOContext,
+ * IOObject, and IOOp which are not tracked for the passed-in BackendType. The
+ * passed-in PgStat_BackendIO must contain stats from the BackendType specified
+ * by the second parameter. Caller is responsible for locking the passed-in
+ * PgStat_BackendIO, if needed.
+ */
+bool
+pgstat_bktype_io_stats_valid(PgStat_BackendIO *backend_io,
+ BackendType bktype)
+{
+ bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ /*
+ * Don't bother trying to skip to the next loop iteration if
+ * pgstat_tracks_io_object() would return false here. We still
+ * need to validate that each counter is zero anyway.
+ */
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ if ((!bktype_tracked || !pgstat_tracks_io_op(bktype, io_context, io_object, io_op)) &&
+ backend_io->data[io_context][io_object][io_op] != 0)
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 2e20b93c20..f793ac1516 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -206,7 +206,7 @@ pgstat_drop_relation(Relation rel)
}
/*
- * Report that the table was just vacuumed.
+ * Report that the table was just vacuumed and flush IO statistics.
*/
void
pgstat_report_vacuum(Oid tableoid, bool shared,
@@ -258,10 +258,18 @@ pgstat_report_vacuum(Oid tableoid, bool shared,
}
pgstat_unlock_entry(entry_ref);
+
+ /*
+ * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
+ * however this will not be called until after an entire autovacuum cycle
+ * is done -- which will likely vacuum many relations -- or until the
+ * VACUUM command has processed all tables and committed.
+ */
+ pgstat_flush_io(false);
}
/*
- * Report that the table was just analyzed.
+ * Report that the table was just analyzed and flush IO statistics.
*
* Caller must provide new live- and dead-tuples estimates, as well as a
* flag indicating whether to reset the mod_since_analyze counter.
@@ -341,6 +349,9 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
+
+ /* see pgstat_report_vacuum() */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index c1506b53d0..09fffd0e82 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -202,6 +202,10 @@ StatsShmemInit(void)
LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&ctl->io.locks[i],
+ LWTRANCHE_PGSTATS_DATA);
}
else
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index e7a82b5fed..e8598b2f4e 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -34,7 +34,7 @@ static WalUsage prevWalUsage;
/*
* Calculate how much WAL usage counters have increased and update
- * shared statistics.
+ * shared WAL and IO statistics.
*
* Must be called by processes that generate WAL, that do not call
* pgstat_report_stat(), like walwriter.
@@ -43,6 +43,8 @@ void
pgstat_report_wal(bool force)
{
pgstat_flush_wal(force);
+
+ pgstat_flush_io(force);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 58bd1360b9..5c8e8336bf 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1593,6 +1593,8 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_BGWRITER);
pgstat_reset_of_kind(PGSTAT_KIND_CHECKPOINTER);
}
+ else if (strcmp(target, "io") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_IO);
else if (strcmp(target, "recovery_prefetch") == 0)
XLogPrefetchResetStats();
else if (strcmp(target, "wal") == 0)
@@ -1601,7 +1603,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"io\", \"recovery_prefetch\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 96b3a1e1a0..c309e0233d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -332,6 +332,8 @@ typedef enum BackendType
B_WAL_WRITER,
} BackendType;
+#define BACKEND_NUM_TYPES (B_WAL_WRITER + 1)
+
extern PGDLLIMPORT BackendType MyBackendType;
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5e3326a3b9..f2a66ed7fb 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -48,6 +48,7 @@ typedef enum PgStat_Kind
PGSTAT_KIND_ARCHIVER,
PGSTAT_KIND_BGWRITER,
PGSTAT_KIND_CHECKPOINTER,
+ PGSTAT_KIND_IO,
PGSTAT_KIND_SLRU,
PGSTAT_KIND_WAL,
} PgStat_Kind;
@@ -276,6 +277,55 @@ typedef struct PgStat_CheckpointerStats
PgStat_Counter buf_fsync_backend;
} PgStat_CheckpointerStats;
+
+/*
+ * Types related to counting IO operations
+ */
+typedef enum IOContext
+{
+ IOCONTEXT_BULKREAD,
+ IOCONTEXT_BULKWRITE,
+ IOCONTEXT_NORMAL,
+ IOCONTEXT_VACUUM,
+} IOContext;
+
+#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
+#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
+
+typedef enum IOObject
+{
+ IOOBJECT_RELATION,
+ IOOBJECT_TEMP_RELATION,
+} IOObject;
+
+#define IOOBJECT_FIRST IOOBJECT_RELATION
+#define IOOBJECT_NUM_TYPES (IOOBJECT_TEMP_RELATION + 1)
+
+typedef enum IOOp
+{
+ IOOP_EVICT,
+ IOOP_EXTEND,
+ IOOP_FSYNC,
+ IOOP_READ,
+ IOOP_REUSE,
+ IOOP_WRITE,
+} IOOp;
+
+#define IOOP_FIRST IOOP_EVICT
+#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
+
+typedef struct PgStat_BackendIO
+{
+ PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
+} PgStat_BackendIO;
+
+typedef struct PgStat_IO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BackendIO stats[BACKEND_NUM_TYPES];
+} PgStat_IO;
+
+
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -453,6 +503,22 @@ extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern void pgstat_count_io_op(IOOp io_op, IOObject io_object, IOContext io_context);
+extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern const char *pgstat_get_io_context_name(IOContext io_context);
+extern const char *pgstat_get_io_object_name(IOObject io_object);
+
+extern bool pgstat_tracks_io_bktype(BackendType bktype);
+extern bool pgstat_tracks_io_object(BackendType bktype,
+ IOContext io_context, IOObject io_object);
+extern bool pgstat_tracks_io_op(BackendType bktype, IOContext io_context,
+ IOObject io_object, IOOp io_op);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 12fd51f1ae..bf8e4c3b8b 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -329,6 +329,17 @@ typedef struct PgStatShared_Checkpointer
PgStat_CheckpointerStats reset_offset;
} PgStatShared_Checkpointer;
+/* shared version of PgStat_IO */
+typedef struct PgStatShared_IO
+{
+ /*
+ * locks[i] protects stats.stats[i]. locks[0] also protects
+ * stats.stat_reset_timestamp.
+ */
+ LWLock locks[BACKEND_NUM_TYPES];
+ PgStat_IO stats;
+} PgStatShared_IO;
+
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -419,6 +430,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Archiver archiver;
PgStatShared_BgWriter bgwriter;
PgStatShared_Checkpointer checkpointer;
+ PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
} PgStat_ShmemControl;
@@ -442,6 +454,8 @@ typedef struct PgStat_Snapshot
PgStat_CheckpointerStats checkpointer;
+ PgStat_IO io;
+
PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS];
PgStat_WalStats wal;
@@ -549,6 +563,17 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern void pgstat_io_reset_all_cb(TimestampTz ts);
+extern void pgstat_io_snapshot_cb(void);
+extern bool pgstat_flush_io(bool nowait);
+extern bool pgstat_bktype_io_stats_valid(PgStat_BackendIO *context_ops,
+ BackendType bktype);
+
+
/*
* Functions in pgstat_relation.c
*/
@@ -643,6 +668,13 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid)
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
+/*
+ * Variables in pgstat_io.c
+ */
+
+extern PGDLLIMPORT bool have_iostats;
+
+
/*
* Variables in pgstat_slru.c
*/
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 23bafec5f7..7b66b1bc89 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1106,7 +1106,10 @@ ID
INFIX
INT128
INTERFACE_INFO
+IOContext
IOFuncSelector
+IOObject
+IOOp
IPCompareMethod
ITEM
IV
@@ -2016,6 +2019,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2033,6 +2037,8 @@ PgStat_FetchConsistency
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
+PgStat_IO
+PgStat_BackendIO
PgStat_Kind
PgStat_KindInfo
PgStat_LocalState
--
2.34.1
[text/x-patch] v47-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch (34.0K, ../../CAAKRu_bHwGEbzNxxy+MQDkrsgog6aO6iUvajJ4d6PD98gFU7+w@mail.gmail.com/4-v47-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch)
download | inline diff:
From b112a4114b9b61b19e6a62ed5fc82a01d8b1d1a2 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 9 Jan 2023 14:42:25 -0500
Subject: [PATCH v47 4/5] Add system view tracking IO ops per backend type
Add pg_stat_io, a system view which tracks the number of IOOps
(evictions, reuses, reads, writes, extensions, and fsyncs) done on each
IOObject (relation, temp relation) in each IOContext ("normal" and those
using a BufferAccessStrategy) by each type of backend (e.g. client
backend, checkpointer).
Some BackendTypes do not accumulate IO operations statistics and will
not be included in the view.
Some IOContexts are not used by some BackendTypes and will not be in the
view. For example, checkpointer does not use a BufferAccessStrategy
(currently), so there will be no rows for BufferAccessStrategy
IOContexts for checkpointer.
Some IOObjects are never operated on in some IOContexts or by some
BackendTypes. These rows are omitted from the view. For example,
checkpointer will never operate on IOOBJECT_TEMP_RELATION data, so those
rows are omitted.
Some IOOps are invalid in combination with certain IOContexts and
certain IOObjects. Those cells will be NULL in the view to distinguish
between 0 observed IOOps of that type and an invalid combination. For
example, temporary tables are not fsynced so cells for all BackendTypes
for IOOBJECT_TEMP_RELATION and IOOP_FSYNC will be NULL.
Some BackendTypes never perform certain IOOps. Those cells will also be
NULL in the view. For example, bgwriter should not perform reads.
View stats are populated with statistics incremented when a backend
performs an IO Operation and maintained by the cumulative statistics
subsystem.
Each row of the view shows stats for a particular BackendType, IOObject,
IOContext combination (e.g. a client backend's operations on permanent
relations in shared buffers) and each column in the view is the total
number of IO Operations done (e.g. writes). So a cell in the view would
be, for example, the number of blocks of relation data written from
shared buffers by client backends since the last stats reset.
In anticipation of tracking WAL IO and non-block-oriented IO (such as
temporary file IO), the "op_bytes" column specifies the unit of the
"reads", "writes", and "extends" columns for a given row.
Note that some of the cells in the view are redundant with fields in
pg_stat_bgwriter (e.g. buffers_backend), however these have been kept in
pg_stat_bgwriter for backwards compatibility. Deriving the redundant
pg_stat_bgwriter stats from the IO operations stats structures was also
problematic due to the separate reset targets for 'bgwriter' and 'io'.
Suggested by Andres Freund
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
contrib/amcheck/expected/check_heap.out | 31 +++
contrib/amcheck/sql/check_heap.sql | 24 +++
src/backend/catalog/system_views.sql | 15 ++
src/backend/utils/adt/pgstatfuncs.c | 154 +++++++++++++++
src/include/catalog/pg_proc.dat | 9 +
src/test/regress/expected/rules.out | 12 ++
src/test/regress/expected/stats.out | 246 ++++++++++++++++++++++++
src/test/regress/sql/stats.sql | 150 +++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 642 insertions(+)
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index c010361025..055107f6b5 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -66,6 +66,19 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy. This allows
+-- us to reliably test that pg_stat_io BULKREAD reads are being captured
+-- without relying on the size of shared buffers or on an expensive operation
+-- like CREATE DATABASE.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -88,6 +101,23 @@ SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock :=
-------+--------+--------+-----
(0 rows)
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+ ?column?
+----------
+ t
+(1 row)
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
SET ROLE regress_heaptest_role;
@@ -195,6 +225,7 @@ ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 298de6886a..1cfd52bd13 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -20,11 +20,26 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
+
-- Add some data so subsequent tests are not entirely trivial
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy. This allows
+-- us to reliably test that pg_stat_io BULKREAD reads are being captured
+-- without relying on the size of shared buffers or on an expensive operation
+-- like CREATE DATABASE.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
+
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -32,6 +47,14 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');
SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
@@ -110,6 +133,7 @@ SELECT * FROM verify_heapam('test_foreign_table',
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d2a8c82900..f875742068 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1116,6 +1116,21 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+CREATE VIEW pg_stat_io AS
+SELECT
+ b.backend_type,
+ b.io_context,
+ b.io_object,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+FROM pg_stat_get_io() b;
+
CREATE VIEW pg_stat_wal AS
SELECT
w.wal_records,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5c8e8336bf..6a5a1dc79f 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1234,6 +1234,160 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+* When adding a new column to the pg_stat_io view, add a new enum value
+* here above IO_NUM_COLUMNS.
+*/
+typedef enum io_stat_col
+{
+ IO_COL_BACKEND_TYPE,
+ IO_COL_IO_CONTEXT,
+ IO_COL_IO_OBJECT,
+ IO_COL_READS,
+ IO_COL_WRITES,
+ IO_COL_EXTENDS,
+ IO_COL_CONVERSION,
+ IO_COL_EVICTIONS,
+ IO_COL_REUSES,
+ IO_COL_FSYNCS,
+ IO_COL_RESET_TIME,
+ IO_NUM_COLUMNS,
+} io_stat_col;
+
+/*
+ * When adding a new IOOp, add a new io_stat_col and add a case to this
+ * function returning the corresponding io_stat_col.
+ */
+static io_stat_col
+pgstat_get_io_op_index(IOOp io_op)
+{
+ switch (io_op)
+ {
+ case IOOP_EVICT:
+ return IO_COL_EVICTIONS;
+ case IOOP_READ:
+ return IO_COL_READS;
+ case IOOP_REUSE:
+ return IO_COL_REUSES;
+ case IOOP_WRITE:
+ return IO_COL_WRITES;
+ case IOOP_EXTEND:
+ return IO_COL_EXTENDS;
+ case IOOP_FSYNC:
+ return IO_COL_FSYNCS;
+ }
+
+ elog(ERROR, "unrecognized IOOp value: %d", io_op);
+ pg_unreachable();
+}
+
+#ifdef USE_ASSERT_CHECKING
+static bool
+pgstat_iszero_io_object(const PgStat_Counter *obj)
+{
+ for (IOOp io_op = IOOP_EVICT; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ if (obj[io_op] != 0)
+ return false;
+ }
+
+ return true;
+}
+#endif
+
+Datum
+pg_stat_get_io(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo;
+ PgStat_IO *backends_io_stats;
+ Datum reset_time;
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ backends_io_stats = pgstat_fetch_stat_io();
+
+ reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
+
+ for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ {
+ bool bktype_tracked;
+ Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
+ PgStat_BackendIO *bktype_stats = &backends_io_stats->stats[bktype];
+
+ /*
+ * For those BackendTypes without IO Operation stats, skip
+ * representing them in the view altogether. We still loop through
+ * their counters so that we can assert that all values are zero.
+ */
+ bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ const char *context_name = pgstat_get_io_context_name(io_context);
+
+ for (IOObject io_obj = IOOBJECT_FIRST;
+ io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ {
+ const char *obj_name = pgstat_get_io_object_name(io_obj);
+
+ Datum values[IO_NUM_COLUMNS] = {0};
+ bool nulls[IO_NUM_COLUMNS] = {0};
+
+ /*
+ * Some combinations of IOContext, IOObject, and BackendType
+ * are not valid for any type of IOOp. In such cases, omit the
+ * entire row from the view.
+ */
+ if (!bktype_tracked ||
+ !pgstat_tracks_io_object(bktype, io_context, io_obj))
+ {
+ Assert(pgstat_iszero_io_object(bktype_stats->data[io_context][io_obj]));
+ continue;
+ }
+
+ values[IO_COL_BACKEND_TYPE] = bktype_desc;
+ values[IO_COL_IO_CONTEXT] = CStringGetTextDatum(context_name);
+ values[IO_COL_IO_OBJECT] = CStringGetTextDatum(obj_name);
+ values[IO_COL_RESET_TIME] = TimestampTzGetDatum(reset_time);
+
+ /*
+ * Hard-code this to the value of BLCKSZ for now. Future
+ * values could include XLOG_BLCKSZ, once WAL IO is tracked,
+ * and constant multipliers, once non-block-oriented IO (e.g.
+ * temporary file IO) is tracked.
+ */
+ values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
+
+ /*
+ * Some combinations of BackendType and IOOp, of IOContext and
+ * IOOp, and of IOObject and IOOp are not tracked. Set these
+ * cells in the view NULL and assert that these stats are zero
+ * as expected.
+ */
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ int col_idx = pgstat_get_io_op_index(io_op);
+
+ nulls[col_idx] = !pgstat_tracks_io_op(bktype, io_context, io_obj, io_op);
+
+ if (!nulls[col_idx])
+ values[col_idx] =
+ Int64GetDatum(bktype_stats->data[io_context][io_obj][io_op]);
+ else
+ Assert(bktype_stats->data[io_context][io_obj][io_op] == 0);
+ }
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ }
+ }
+
+ return (Datum) 0;
+}
+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3810de7b22..57a889cf49 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5690,6 +5690,15 @@
proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
+{ oid => '8459', descr => 'statistics: per backend type IO statistics',
+ proname => 'pg_stat_get_io', provolatile => 'v',
+ prorows => '30', proretset => 't',
+ proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,text,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{backend_type,io_context,io_object,reads,writes,extends,op_bytes,evictions,reuses,fsyncs,stats_reset}',
+ prosrc => 'pg_stat_get_io' },
+
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index a969ae63eb..dd5ddffc4d 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1876,6 +1876,18 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_enc AS encrypted
FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
WHERE (s.client_port IS NOT NULL);
+pg_stat_io| SELECT b.backend_type,
+ b.io_context,
+ b.io_object,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+ FROM pg_stat_get_io() b(backend_type, io_context, io_object, reads, writes, extends, op_bytes, evictions, reuses, fsyncs, stats_reset);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 1d84407a03..a66fe86b05 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1126,4 +1126,250 @@ SELECT pg_stat_get_subscription_stats(NULL);
(1 row)
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_shared_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_extends_after > :io_sum_shared_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+-- The second checkpoint ensures that stats from the first checkpoint have been
+-- reported and protects against any potential races amongst the table
+-- creation, a possible timing-triggered checkpoint, and the explicit
+-- checkpoint in the test.
+SELECT sum(writes) AS io_sum_shared_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS io_sum_shared_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_writes_after > :io_sum_shared_writes_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT current_setting('fsync') = 'off' OR :io_sum_shared_fsyncs_after > :io_sum_shared_fsyncs_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+ count
+-------
+ 100
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS io_sum_shared_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_reads_after > :io_sum_shared_reads_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+-- Set temp_buffers to a low value so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO '1MB';
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS io_sum_local_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(evictions) AS io_sum_local_evictions_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 8000) as id, repeat('a', 100);
+SELECT sum(reads) AS io_sum_local_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+ count
+-------
+ 8000
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(evictions) AS io_sum_local_evictions_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(reads) AS io_sum_local_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(extends) AS io_sum_local_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_evictions_after > :io_sum_local_evictions_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_local_reads_after > :io_sum_local_reads_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_local_writes_after > :io_sum_local_writes_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_local_extends_after > :io_sum_local_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(writes) AS io_sum_local_writes_new_tblspc
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_writes_new_tblspc > :io_sum_local_writes_after;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT :io_sum_vac_strategy_reads_after > :io_sum_vac_strategy_reads_before;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT :io_sum_vac_strategy_reuses_after > :io_sum_vac_strategy_reuses_before;
+ ?column?
+----------
+ t
+(1 row)
+
+RESET wal_skip_threshold;
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test IO stats reset
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+ ?column?
+----------
+ t
+(1 row)
+
-- End of Stats Test
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index b4d6753c71..64b3da2765 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -536,4 +536,154 @@ SELECT pg_stat_get_replication_slot(NULL);
SELECT pg_stat_get_subscription_stats(NULL);
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_shared_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_extends_after > :io_sum_shared_extends_before;
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+-- The second checkpoint ensures that stats from the first checkpoint have been
+-- reported and protects against any potential races amongst the table
+-- creation, a possible timing-triggered checkpoint, and the explicit
+-- checkpoint in the test.
+SELECT sum(writes) AS io_sum_shared_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS io_sum_shared_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT sum(fsyncs) AS io_sum_shared_fsyncs_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+
+SELECT :io_sum_shared_writes_after > :io_sum_shared_writes_before;
+SELECT current_setting('fsync') = 'off' OR :io_sum_shared_fsyncs_after > :io_sum_shared_fsyncs_before;
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS io_sum_shared_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_reads_after > :io_sum_shared_reads_before;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+
+-- Set temp_buffers to a low value so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO '1MB';
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS io_sum_local_extends_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(evictions) AS io_sum_local_evictions_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 8000) as id, repeat('a', 100);
+
+SELECT sum(reads) AS io_sum_local_reads_before
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+SELECT pg_stat_force_next_flush();
+SELECT sum(evictions) AS io_sum_local_evictions_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(reads) AS io_sum_local_reads_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(writes) AS io_sum_local_writes_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT sum(extends) AS io_sum_local_extends_after
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_evictions_after > :io_sum_local_evictions_before;
+SELECT :io_sum_local_reads_after > :io_sum_local_reads_before;
+SELECT :io_sum_local_writes_after > :io_sum_local_writes_before;
+SELECT :io_sum_local_extends_after > :io_sum_local_extends_before;
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+SELECT sum(writes) AS io_sum_local_writes_new_tblspc
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_writes_new_tblspc > :io_sum_local_writes_after;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reuses) AS io_sum_vac_strategy_reuses_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT sum(reads) AS io_sum_vac_strategy_reads_after FROM pg_stat_io WHERE io_context = 'vacuum' \gset
+SELECT :io_sum_vac_strategy_reads_after > :io_sum_vac_strategy_reads_before;
+SELECT :io_sum_vac_strategy_reuses_after > :io_sum_vac_strategy_reuses_before;
+RESET wal_skip_threshold;
+
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+
+-- Test IO stats reset
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+
-- End of Stats Test
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7b66b1bc89..c4ecef2bf8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3377,6 +3377,7 @@ intset_internal_node
intset_leaf_node
intset_node
intvKEY
+io_stat_col
itemIdCompact
itemIdCompactData
iterator
--
2.34.1
[text/x-patch] v47-0001-pgindent-and-some-manual-cleanup-in-pgstat-relat.patch (6.0K, ../../CAAKRu_bHwGEbzNxxy+MQDkrsgog6aO6iUvajJ4d6PD98gFU7+w@mail.gmail.com/5-v47-0001-pgindent-and-some-manual-cleanup-in-pgstat-relat.patch)
download | inline diff:
From f8c9077631169a778c893fd16b7a973ad5725f2a Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 9 Dec 2022 18:23:19 -0800
Subject: [PATCH v47 1/5] pgindent and some manual cleanup in pgstat related
code
---
src/backend/storage/buffer/bufmgr.c | 22 ++++++++++----------
src/backend/storage/buffer/localbuf.c | 4 ++--
src/backend/utils/activity/pgstat.c | 3 ++-
src/backend/utils/activity/pgstat_relation.c | 1 +
src/backend/utils/adt/pgstatfuncs.c | 2 +-
src/include/pgstat.h | 1 +
src/include/utils/pgstat_internal.h | 1 +
7 files changed, 19 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 3fb38a25cf..8075828e8a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -516,7 +516,7 @@ PrefetchSharedBuffer(SMgrRelation smgr_reln,
/* create a tag so we can lookup the buffer */
InitBufferTag(&newTag, &smgr_reln->smgr_rlocator.locator,
- forkNum, blockNum);
+ forkNum, blockNum);
/* determine its hash code and partition lock ID */
newHash = BufTableHashCode(&newTag);
@@ -3297,8 +3297,8 @@ DropRelationsAllBuffers(SMgrRelation *smgr_reln, int nlocators)
uint32 buf_state;
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (!use_bsearch)
@@ -3425,8 +3425,8 @@ DropDatabaseBuffers(Oid dbid)
uint32 buf_state;
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (bufHdr->tag.dbOid != dbid)
continue;
@@ -3572,8 +3572,8 @@ FlushRelationBuffers(Relation rel)
bufHdr = GetBufferDescriptor(i);
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (!BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator))
continue;
@@ -3645,8 +3645,8 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
uint32 buf_state;
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (!use_bsearch)
@@ -3880,8 +3880,8 @@ FlushDatabaseBuffers(Oid dbid)
bufHdr = GetBufferDescriptor(i);
/*
- * As in DropRelationBuffers, an unlocked precheck should be
- * safe and saves some cycles.
+ * As in DropRelationBuffers, an unlocked precheck should be safe and
+ * saves some cycles.
*/
if (bufHdr->tag.dbOid != dbid)
continue;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index b2720df6ea..8372acc383 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -610,8 +610,8 @@ AtProcExit_LocalBuffers(void)
{
/*
* We shouldn't be holding any remaining pins; if we are, and assertions
- * aren't enabled, we'll fail later in DropRelationBuffers while
- * trying to drop the temp rels.
+ * aren't enabled, we'll fail later in DropRelationBuffers while trying to
+ * drop the temp rels.
*/
CheckForLocalBufferLeaks();
}
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 7e9dc17e68..0fa5370bcd 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -426,7 +426,7 @@ pgstat_discard_stats(void)
ereport(DEBUG2,
(errcode_for_file_access(),
errmsg_internal("unlinked permanent statistics file \"%s\"",
- PGSTAT_STAT_PERMANENT_FILENAME)));
+ PGSTAT_STAT_PERMANENT_FILENAME)));
}
/*
@@ -986,6 +986,7 @@ pgstat_build_snapshot(void)
entry->data = MemoryContextAlloc(pgStatLocal.snapshot.context,
kind_info->shared_size);
+
/*
* Acquire the LWLock directly instead of using
* pg_stat_lock_entry_shared() which requires a reference.
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 1730425de1..2e20b93c20 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -783,6 +783,7 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
if (lstats->t_counts.t_numscans)
{
TimestampTz t = GetCurrentTransactionStopTimestamp();
+
if (t > tabentry->lastscan)
tabentry->lastscan = t;
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 6cddd74aa7..58bd1360b9 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -906,7 +906,7 @@ pg_stat_get_backend_client_addr(PG_FUNCTION_ARGS)
clean_ipv6_addr(beentry->st_clientaddr.addr.ss_family, remote_host);
PG_RETURN_DATUM(DirectFunctionCall1(inet_in,
- CStringGetDatum(remote_host)));
+ CStringGetDatum(remote_host)));
}
Datum
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index d3e965d744..5e3326a3b9 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -476,6 +476,7 @@ extern void pgstat_report_connect(Oid dboid);
extern PgStat_StatDBEntry *pgstat_fetch_stat_dbentry(Oid dboid);
+
/*
* Functions in pgstat_function.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 08412d6404..12fd51f1ae 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -626,6 +626,7 @@ extern void pgstat_wal_snapshot_cb(void);
extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
+
/*
* Functions in pgstat_xact.c
*/
--
2.34.1
[text/x-patch] v47-0005-pg_stat_io-documentation.patch (13.3K, ../../CAAKRu_bHwGEbzNxxy+MQDkrsgog6aO6iUvajJ4d6PD98gFU7+w@mail.gmail.com/6-v47-0005-pg_stat_io-documentation.patch)
download | inline diff:
From 62beac2b02e1b724f96533d81515b2e9b3d1f5d9 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 9 Jan 2023 14:42:53 -0500
Subject: [PATCH v47 5/5] pg_stat_io documentation
Author: Melanie Plageman <[email protected]>
Author: Samay Sharma <[email protected]>
Reviewed-by: Maciek Sakrejda <[email protected]>
Reviewed-by: Lukas Fittl <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 318 +++++++++++++++++++++++++++++++++--
1 file changed, 304 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 8d51ca3773..ec8dba00f9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -469,6 +469,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
+ <entry>
+ One row for each combination of backend type, context, and target object
+ containing cluster-wide I/O statistics.
+ See <link linkend="monitoring-pg-stat-io-view">
+ <structname>pg_stat_io</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_replication_slots</structname><indexterm><primary>pg_stat_replication_slots</primary></indexterm></entry>
<entry>One row per replication slot, showing statistics about the
@@ -665,20 +675,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</para>
<para>
- The <structname>pg_statio_</structname> views are primarily useful to
- determine the effectiveness of the buffer cache. When the number
- of actual disk reads is much smaller than the number of buffer
- hits, then the cache is satisfying most read requests without
- invoking a kernel call. However, these statistics do not give the
- entire story: due to the way in which <productname>PostgreSQL</productname>
- handles disk I/O, data that is not in the
- <productname>PostgreSQL</productname> buffer cache might still reside in the
- kernel's I/O cache, and might therefore still be fetched without
- requiring a physical read. Users interested in obtaining more
- detailed information on <productname>PostgreSQL</productname> I/O behavior are
- advised to use the <productname>PostgreSQL</productname> statistics views
- in combination with operating system utilities that allow insight
- into the kernel's handling of I/O.
+ The <structname>pg_stat_io</structname> and
+ <structname>pg_statio_</structname> set of views are useful for determining
+ the effectiveness of the buffer cache. They can be used to calculate a cache
+ hit ratio. Note that while <productname>PostgreSQL</productname>'s I/O
+ statistics capture most instances in which the kernel was invoked in order
+ to perform I/O, they do not differentiate between data which had to be
+ fetched from disk and that which already resided in the kernel page cache.
+ Users are advised to use the <productname>PostgreSQL</productname>
+ statistics views in combination with operating system utilities for a more
+ complete picture of their database's I/O performance.
</para>
</sect2>
@@ -3643,6 +3649,290 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_archived_wal</structfield> have also been successfully
archived.
</para>
+ </sect2>
+
+ <sect2 id="monitoring-pg-stat-io-view">
+ <title><structname>pg_stat_io</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_io</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_io</structname> view will contain one row for each
+ combination of backend type, I/O context, and target I/O object showing
+ cluster-wide I/O statistics. Combinations which do not make sense are
+ omitted.
+ </para>
+
+ <para>
+ Currently, I/O on relations (e.g. tables, indexes) is tracked. However,
+ relation I/O which bypasses shared buffers (e.g. when moving a table from one
+ tablespace to another) is currently not tracked.
+ </para>
+
+ <table id="pg-stat-io-view" xreflabel="pg_stat_io">
+ <title><structname>pg_stat_io</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>backend_type</structfield> <type>text</type>
+ </para>
+ <para>
+ Type of backend (e.g. background worker, autovacuum worker). See <link
+ linkend="monitoring-pg-stat-activity-view">
+ <structname>pg_stat_activity</structname></link> for more information
+ on <varname>backend_type</varname>s. Some
+ <varname>backend_type</varname>s do not accumulate I/O operation
+ statistics and will not be included in the view.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_context</structfield> <type>text</type>
+ </para>
+ <para>
+ The context of an I/O operation. Possible values are:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>normal</literal>: The default or standard
+ <varname>io_context</varname> for a type of I/O operation. For
+ example, by default, relation data is read into and written out from
+ shared buffers. Thus, reads and writes of relation data to and from
+ shared buffers are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>vacuum</literal>: I/O operations performed outside of shared
+ buffers while vacuuming and analyzing permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkread</literal>: Certain large read I/O operations
+ done outside of shared buffers, for example, a sequential scan of a
+ large table.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkwrite</literal>: Certain large write I/O operations
+ done outside of shared buffers, such as <command>COPY</command>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_object</structfield> <type>text</type>
+ </para>
+ <para>
+ Target object of an I/O operation. Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>relation</literal>: Permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>temp relation</literal>: Temporary relations.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reads</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of read operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>writes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of write operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>extends</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of relation extend operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>op_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of bytes per unit of I/O read, written, or extended.
+ </para>
+ <para>
+ Relation data reads, writes, and extends are done in
+ <varname>block_size</varname> units, derived from the build-time
+ parameter <symbol>BLCKSZ</symbol>, which is <literal>8192</literal> by
+ default.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a block has been written out from a shared or local
+ buffer in order to make it available for another use.
+ </para>
+ <para>
+ In <varname>io_context</varname> <literal>normal</literal>, this counts
+ the number of times a block was evicted from a buffer and replaced with
+ another block. In <varname>io_context</varname>s
+ <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
+ <literal>vacuum</literal>, this counts the number of times a block was
+ evicted from shared buffers in order to add the shared buffer to a
+ separate, size-limited ring buffer for use in a bulk I/O operation.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reuses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of times an existing buffer in a size-limited ring buffer
+ outside of shared buffers was reused as part of an I/O operation in the
+ <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
+ <literal>vacuum</literal> <varname>io_context</varname>s.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>fsyncs</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of <literal>fsync</literal> calls. These are only tracked in
+ <varname>io_context</varname> <literal>normal</literal>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset.
+ </para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Some backend types never perform I/O operations in some I/O contexts and/or
+ on some I/O objects. These rows are omitted from the view. For example, the
+ checkpointer does not checkpoint temporary tables, so there will be no rows
+ for <varname>backend_type</varname> <literal>checkpointer</literal> and
+ <varname>io_object</varname> <literal>temp relation</literal>.
+ </para>
+
+ <para>
+ In addition, some I/O operations will never be performed either by certain
+ backend types or in certain I/O contexts or on certain I/O objects. These
+ cells will be NULL. For example, temporary tables are not
+ <literal>fsync</literal>ed, so <varname>fsyncs</varname> will be NULL for
+ <varname>io_object</varname> <literal>temp relation</literal>. Also, the
+ background writer does not perform reads, so <varname>reads</varname> will
+ be NULL in rows for <varname>backend_type</varname> <literal>background
+ writer</literal>.
+ </para>
+
+ <para>
+ <structname>pg_stat_io</structname> can be used to inform database tuning.
+ For example:
+ <itemizedlist>
+ <listitem>
+ <para>
+ A high <varname>evictions</varname> count can indicate that shared
+ buffers should be increased.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Client backends rely on the checkpointer to ensure data is persisted to
+ permanent storage. Large numbers of <varname>fsyncs</varname> by
+ <literal>client backend</literal>s could indicate a misconfiguration of
+ shared buffers or of the checkpointer. More information on configuring
+ the checkpointer can be found in <xref linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Normally, client backends should be able to rely on auxiliary processes
+ like the checkpointer and the background writer to write out dirty data
+ as much as possible. Large numbers of writes by client backends could
+ indicate a misconfiguration of shared buffers or of the checkpointer.
+ More information on configuring the checkpointer can be found in <xref
+ linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
</sect2>
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-13 23:36 ` Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Andres Freund @ 2023-01-13 23:36 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; vignesh C <[email protected]>
Hi,
On 2023-01-13 13:38:15 -0500, Melanie Plageman wrote:
> > I think that's marginally better, but I think having to define both
> > FIRST and NUM is excessive and doesn't make it less fragile. Not sure
> > what anyone else will say, but I'd prefer if it started at "0".
The reason for using FIRST is to be able to define the loop variable as the
enum type, without assigning numeric values to an enum var. I prefer it
slightly.
> From f8c9077631169a778c893fd16b7a973ad5725f2a Mon Sep 17 00:00:00 2001
> From: Andres Freund <[email protected]>
> Date: Fri, 9 Dec 2022 18:23:19 -0800
> Subject: [PATCH v47 1/5] pgindent and some manual cleanup in pgstat related
Applied.
> Subject: [PATCH v47 2/5] pgstat: Infrastructure to track IO operations
> diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
> index 0fa5370bcd..608c3b59da 100644
> --- a/src/backend/utils/activity/pgstat.c
> +++ b/src/backend/utils/activity/pgstat.c
Reminder to self: Need to bump PGSTAT_FILE_FORMAT_ID before commit.
Perhaps you could add a note about that to the commit message?
> @@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
> .snapshot_cb = pgstat_checkpointer_snapshot_cb,
> },
>
> + [PGSTAT_KIND_IO] = {
> + .name = "io_ops",
That should be "io" now I think?
> +/*
> + * Check that stats have not been counted for any combination of IOContext,
> + * IOObject, and IOOp which are not tracked for the passed-in BackendType. The
> + * passed-in PgStat_BackendIO must contain stats from the BackendType specified
> + * by the second parameter. Caller is responsible for locking the passed-in
> + * PgStat_BackendIO, if needed.
> + */
Other PgStat_Backend* structs are just for pending data. Perhaps we could
rename it slightly to make that clearer? PgStat_BktypeIO?
PgStat_IOForBackendType? or a similar variation?
> +bool
> +pgstat_bktype_io_stats_valid(PgStat_BackendIO *backend_io,
> + BackendType bktype)
> +{
> + bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
> +
> + for (IOContext io_context = IOCONTEXT_FIRST;
> + io_context < IOCONTEXT_NUM_TYPES; io_context++)
> + {
> + for (IOObject io_object = IOOBJECT_FIRST;
> + io_object < IOOBJECT_NUM_TYPES; io_object++)
> + {
> + /*
> + * Don't bother trying to skip to the next loop iteration if
> + * pgstat_tracks_io_object() would return false here. We still
> + * need to validate that each counter is zero anyway.
> + */
> + for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
> + {
> + if ((!bktype_tracked || !pgstat_tracks_io_op(bktype, io_context, io_object, io_op)) &&
> + backend_io->data[io_context][io_object][io_op] != 0)
> + return false;
Hm, perhaps this could be broken up into multiple lines? Something like
/* no stats, so nothing to validate */
if (backend_io->data[io_context][io_object][io_op] == 0)
continue;
/* something went wrong if have stats for something not tracked */
if (!bktype_tracked ||
!pgstat_tracks_io_op(bktype, io_context, io_object, io_op))
return false;
> +typedef struct PgStat_BackendIO
> +{
> + PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
> +} PgStat_BackendIO;
Would it bother you if we swapped the order of iocontext and iobject here and
related places? It makes more sense to me semantically, and should now be
pretty easy, code wise.
> +/* shared version of PgStat_IO */
> +typedef struct PgStatShared_IO
> +{
Maybe /* PgStat_IO in shared memory */?
> Subject: [PATCH v47 3/5] pgstat: Count IO for relations
Nearly happy with this now. See one minor nit below.
I don't love the counting in register_dirty_segment() and mdsyncfiletag(), but
I don't have a better idea, and it doesn't seem too horrible.
> @@ -1441,6 +1474,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
>
> UnlockBufHdr(buf, buf_state);
>
> + if (oldFlags & BM_VALID)
> + {
> + /*
> + * When a BufferAccessStrategy is in use, blocks evicted from shared
> + * buffers are counted as IOOP_EVICT in the corresponding context
> + * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
> + * strategy in two cases: 1) while initially claiming buffers for the
> + * strategy ring 2) to replace an existing strategy ring buffer
> + * because it is pinned or in use and cannot be reused.
> + *
> + * Blocks evicted from buffers already in the strategy ring are
> + * counted as IOOP_REUSE in the corresponding strategy context.
> + *
> + * At this point, we can accurately count evictions and reuses,
> + * because we have successfully claimed the valid buffer. Previously,
> + * we may have been forced to release the buffer due to concurrent
> + * pinners or erroring out.
> + */
> + pgstat_count_io_op(from_ring ? IOOP_REUSE : IOOP_EVICT,
> + IOOBJECT_RELATION, *io_context);
> + }
> +
> if (oldPartitionLock != NULL)
> {
> BufTableDelete(&oldTag, oldHash);
There's no reason to do this while we still hold the buffer partition lock,
right? That's a highly contended lock, and we can just move the counting a few
lines down.
> @@ -1410,6 +1432,9 @@ mdsyncfiletag(const FileTag *ftag, char *path)
> if (need_to_close)
> FileClose(file);
>
> + if (result >= 0)
> + pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
> +
I'd lean towards doing this unconditionally, it's still an fsync if it
failed... Not that it matters.
> Subject: [PATCH v47 4/5] Add system view tracking IO ops per backend type
Note to self + commit message: Remember the need to do a catversion bump.
> +-- pg_stat_io test:
> +-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy.
Maybe add that "whereas a sequential scan does not, see ..."?
> This allows
> +-- us to reliably test that pg_stat_io BULKREAD reads are being captured
> +-- without relying on the size of shared buffers or on an expensive operation
> +-- like CREATE DATABASE.
CREATE / DROP TABLESPACE is also pretty expensive, but I don't have a better
idea.
> +-- Create an alternative tablespace and move the heaptest table to it, causing
> +-- it to be rewritten.
IIRC the point of that is that it reliably evicts all the buffers from s_b,
correct? If so, mention that?
> +Datum
> +pg_stat_get_io(PG_FUNCTION_ARGS)
> +{
> + ReturnSetInfo *rsinfo;
> + PgStat_IO *backends_io_stats;
> + Datum reset_time;
> +
> + InitMaterializedSRF(fcinfo, 0);
> + rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
> +
> + backends_io_stats = pgstat_fetch_stat_io();
> +
> + reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
> +
> + for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
> + {
> + bool bktype_tracked;
> + Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
> + PgStat_BackendIO *bktype_stats = &backends_io_stats->stats[bktype];
> +
> + /*
> + * For those BackendTypes without IO Operation stats, skip
> + * representing them in the view altogether. We still loop through
> + * their counters so that we can assert that all values are zero.
> + */
> + bktype_tracked = pgstat_tracks_io_bktype(bktype);
How about instead just doing Assert(pgstat_bktype_io_stats_valid(...))? That
deduplicates the logic for the asserts, and avoids doing the full loop when
assertions aren't enabled anyway?
Otherwise, see also the suggestion aout formatting the assertions as I
suggested for 0002.
> +-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
> +-- and fsyncs.
> +-- The second checkpoint ensures that stats from the first checkpoint have been
> +-- reported and protects against any potential races amongst the table
> +-- creation, a possible timing-triggered checkpoint, and the explicit
> +-- checkpoint in the test.
There's a comment about the subsequent checkpoints earlier in the file, and I
think the comment is slightly more precise. Mybe just reference the earlier comment?
> +-- Change the tablespace so that the table is rewritten directly, then SELECT
> +-- from it to cause it to be read back into shared buffers.
> +SET allow_in_place_tablespaces = true;
> +CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
Perhaps worth doing this in tablespace.sql, to avoid the additional
checkpoints done as part of CREATE/DROP TABLESPACE?
Or, at least combine this with the CHECKPOINTs above?
> +-- Drop the table so we can drop the tablespace later.
> +DROP TABLE test_io_shared;
> +-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
> +-- - eviction of local buffers in order to reuse them
> +-- - reads of temporary table blocks into local buffers
> +-- - writes of local buffers to permanent storage
> +-- - extends of temporary tables
> +-- Set temp_buffers to a low value so that we can trigger writes with fewer
> +-- inserted tuples. Do so in a new session in case temporary tables have been
> +-- accessed by previous tests in this session.
> +\c
> +SET temp_buffers TO '1MB';
I'd set it to the actual minimum '100' (in pages). Perhaps that'd allow to
make test_io_local a bit smaller?
> +CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
> +SELECT sum(extends) AS io_sum_local_extends_before
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> +SELECT sum(evictions) AS io_sum_local_evictions_before
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> +SELECT sum(writes) AS io_sum_local_writes_before
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> +-- Insert tuples into the temporary table, generating extends in the stats.
> +-- Insert enough values that we need to reuse and write out dirty local
> +-- buffers, generating evictions and writes.
> +INSERT INTO test_io_local SELECT generate_series(1, 8000) as id, repeat('a', 100);
> +SELECT sum(reads) AS io_sum_local_reads_before
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
Maybe add something like
SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
Better toast compression or such could easily make test_io_local smaller than
it's today. Seeing that it's too small would make it easier to understand the
failure.
> +SELECT sum(evictions) AS io_sum_local_evictions_after
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> +SELECT sum(reads) AS io_sum_local_reads_after
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> +SELECT sum(writes) AS io_sum_local_writes_after
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> +SELECT sum(extends) AS io_sum_local_extends_after
> + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
This could just be one select with multiple columns?
I think if you use something like \gset io_sum_local_after_ you can also avoid
the need to repeat "io_sum_local_" so many times.
> +SELECT :io_sum_local_evictions_after > :io_sum_local_evictions_before;
> + ?column?
> +----------
> + t
> +(1 row)
> +
> +SELECT :io_sum_local_reads_after > :io_sum_local_reads_before;
> + ?column?
> +----------
> + t
> +(1 row)
> +
> +SELECT :io_sum_local_writes_after > :io_sum_local_writes_before;
> + ?column?
> +----------
> + t
> +(1 row)
> +
> +SELECT :io_sum_local_extends_after > :io_sum_local_extends_before;
> + ?column?
> +----------
> + t
> +(1 row)
Similar.
> +SELECT sum(reuses) AS io_sum_vac_strategy_reuses_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
> +SELECT sum(reads) AS io_sum_vac_strategy_reads_before FROM pg_stat_io WHERE io_context = 'vacuum' \gset
There's quite a few more instances of this, so I'll now omit further mentions.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-01-17 17:22 ` Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-18 06:10 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Maciek Sakrejda <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: Melanie Plageman @ 2023-01-17 17:22 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; vignesh C <[email protected]>
v48 attached.
On Fri, Jan 13, 2023 at 6:36 PM Andres Freund <[email protected]> wrote:
> On 2023-01-13 13:38:15 -0500, Melanie Plageman wrote:
> > From f8c9077631169a778c893fd16b7a973ad5725f2a Mon Sep 17 00:00:00 2001
> > From: Andres Freund <[email protected]>
> > Date: Fri, 9 Dec 2022 18:23:19 -0800
> > Subject: [PATCH v47 2/5] pgstat: Infrastructure to track IO operations
> > diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
> > index 0fa5370bcd..608c3b59da 100644
> > --- a/src/backend/utils/activity/pgstat.c
> > +++ b/src/backend/utils/activity/pgstat.c
>
> Reminder to self: Need to bump PGSTAT_FILE_FORMAT_ID before commit.
>
> Perhaps you could add a note about that to the commit message?
>
done
>
>
> > @@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
> > .snapshot_cb = pgstat_checkpointer_snapshot_cb,
> > },
> >
> > + [PGSTAT_KIND_IO] = {
> > + .name = "io_ops",
>
> That should be "io" now I think?
>
Oh no! I didn't notice this was broken. I've added pg_stat_have_stats()
to the IO stats tests now.
It would be nice if pgstat_get_kind_from_str() could be used in
pg_stat_reset_shared() to avoid having to remember to change both. It
doesn't really work because we want to be able to throw the error
message in pg_stat_reset_shared() when the user input is wrong -- not
the one in pgstat_get_kind_from_str().
Also:
- Since recovery_prefetch doesn't have a statistic kind, it doesn't fit
well into this paradigm
- Only a subset of the statistics kinds are reset through this function
- bgwriter and checkpointer share a reset target
I added a comment -- perhaps that's all I can do?
On a separate note, should we be setting have_[io/slru/etc]stats to
false in the reset all functions?
>
> > +/*
> > + * Check that stats have not been counted for any combination of IOContext,
> > + * IOObject, and IOOp which are not tracked for the passed-in BackendType. The
> > + * passed-in PgStat_BackendIO must contain stats from the BackendType specified
> > + * by the second parameter. Caller is responsible for locking the passed-in
> > + * PgStat_BackendIO, if needed.
> > + */
>
> Other PgStat_Backend* structs are just for pending data. Perhaps we could
> rename it slightly to make that clearer? PgStat_BktypeIO?
> PgStat_IOForBackendType? or a similar variation?
I've done this.
>
> > +bool
> > +pgstat_bktype_io_stats_valid(PgStat_BackendIO *backend_io,
> > + BackendType bktype)
> > +{
> > + bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
> > +
> > + for (IOContext io_context = IOCONTEXT_FIRST;
> > + io_context < IOCONTEXT_NUM_TYPES; io_context++)
> > + {
> > + for (IOObject io_object = IOOBJECT_FIRST;
> > + io_object < IOOBJECT_NUM_TYPES; io_object++)
> > + {
> > + /*
> > + * Don't bother trying to skip to the next loop iteration if
> > + * pgstat_tracks_io_object() would return false here. We still
> > + * need to validate that each counter is zero anyway.
> > + */
> > + for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
> > + {
> > + if ((!bktype_tracked || !pgstat_tracks_io_op(bktype, io_context, io_object, io_op)) &&
> > + backend_io->data[io_context][io_object][io_op] != 0)
> > + return false;
>
> Hm, perhaps this could be broken up into multiple lines? Something like
>
> /* no stats, so nothing to validate */
> if (backend_io->data[io_context][io_object][io_op] == 0)
> continue;
>
> /* something went wrong if have stats for something not tracked */
> if (!bktype_tracked ||
> !pgstat_tracks_io_op(bktype, io_context, io_object, io_op))
> return false;
I've done this.
> > +typedef struct PgStat_BackendIO
> > +{
> > + PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
> > +} PgStat_BackendIO;
>
> Would it bother you if we swapped the order of iocontext and iobject here and
> related places? It makes more sense to me semantically, and should now be
> pretty easy, code wise.
So, thinking about this I started noticing inconsistencies in other
areas around this order:
For example: ordering of objects mentioned in commit messages and comments,
ordering of parameters (like in pgstat_count_io_op() [currently in
reverse order]).
I think we should make a final decision about this ordering and then
make everywhere consistent (including ordering in the view).
Currently the order is:
BackendType
IOContext
IOObject
IOOp
You are suggesting this order:
BackendType
IOObject
IOContext
IOOp
Could you explain what you find more natural about this ordering (as I
find the other more natural)?
This is one possible natural sentence with these objects:
During COPY, a client backend may read in data from a permanent
relation.
This order is:
IOContext
BackendType
IOOp
IOObject
I think English sentences are often structured subject, verb, object --
but in our case, we have an extra thing that doesn't fit neatly
(IOContext). Also, IOOp in a sentence would be in the middle (as the
verb). I made it last because a) it feels like the smallest unit b) it
would make the code a lot more annoying if it wasn't last.
WRT IOObject and IOContext, is there a future case for which having
IOObject first will be better or lead to fewer mistakes?
I actually see loads of places where this needs to be made consistent.
>
> > +/* shared version of PgStat_IO */
> > +typedef struct PgStatShared_IO
> > +{
>
> Maybe /* PgStat_IO in shared memory */?
>
updated.
>
> > Subject: [PATCH v47 3/5] pgstat: Count IO for relations
>
> Nearly happy with this now. See one minor nit below.
>
> I don't love the counting in register_dirty_segment() and mdsyncfiletag(), but
> I don't have a better idea, and it doesn't seem too horrible.
You don't like it because such things shouldn't be in md.c -- since we
went to the trouble of having function pointers and making it general?
>
> > @@ -1441,6 +1474,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
> >
> > UnlockBufHdr(buf, buf_state);
> >
> > + if (oldFlags & BM_VALID)
> > + {
> > + /*
> > + * When a BufferAccessStrategy is in use, blocks evicted from shared
> > + * buffers are counted as IOOP_EVICT in the corresponding context
> > + * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
> > + * strategy in two cases: 1) while initially claiming buffers for the
> > + * strategy ring 2) to replace an existing strategy ring buffer
> > + * because it is pinned or in use and cannot be reused.
> > + *
> > + * Blocks evicted from buffers already in the strategy ring are
> > + * counted as IOOP_REUSE in the corresponding strategy context.
> > + *
> > + * At this point, we can accurately count evictions and reuses,
> > + * because we have successfully claimed the valid buffer. Previously,
> > + * we may have been forced to release the buffer due to concurrent
> > + * pinners or erroring out.
> > + */
> > + pgstat_count_io_op(from_ring ? IOOP_REUSE : IOOP_EVICT,
> > + IOOBJECT_RELATION, *io_context);
> > + }
> > +
> > if (oldPartitionLock != NULL)
> > {
> > BufTableDelete(&oldTag, oldHash);
>
> There's no reason to do this while we still hold the buffer partition lock,
> right? That's a highly contended lock, and we can just move the counting a few
> lines down.
Thanks, I've done this.
>
> > @@ -1410,6 +1432,9 @@ mdsyncfiletag(const FileTag *ftag, char *path)
> > if (need_to_close)
> > FileClose(file);
> >
> > + if (result >= 0)
> > + pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
> > +
>
> I'd lean towards doing this unconditionally, it's still an fsync if it
> failed... Not that it matters.
Good point. We still incurred the costs if not benefited from the
effects. I've updated this.
>
> > Subject: [PATCH v47 4/5] Add system view tracking IO ops per backend type
>
> Note to self + commit message: Remember the need to do a catversion bump.
Noted.
>
> > +-- pg_stat_io test:
> > +-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy.
>
> Maybe add that "whereas a sequential scan does not, see ..."?
Updated.
>
> > This allows
> > +-- us to reliably test that pg_stat_io BULKREAD reads are being captured
> > +-- without relying on the size of shared buffers or on an expensive operation
> > +-- like CREATE DATABASE.
>
> CREATE / DROP TABLESPACE is also pretty expensive, but I don't have a better
> idea.
I've added a comment.
>
> > +-- Create an alternative tablespace and move the heaptest table to it, causing
> > +-- it to be rewritten.
>
> IIRC the point of that is that it reliably evicts all the buffers from s_b,
> correct? If so, mention that?
Done.
>
> > +Datum
> > +pg_stat_get_io(PG_FUNCTION_ARGS)
> > +{
> > + ReturnSetInfo *rsinfo;
> > + PgStat_IO *backends_io_stats;
> > + Datum reset_time;
> > +
> > + InitMaterializedSRF(fcinfo, 0);
> > + rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
> > +
> > + backends_io_stats = pgstat_fetch_stat_io();
> > +
> > + reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
> > +
> > + for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
> > + {
> > + bool bktype_tracked;
> > + Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
> > + PgStat_BackendIO *bktype_stats = &backends_io_stats->stats[bktype];
> > +
> > + /*
> > + * For those BackendTypes without IO Operation stats, skip
> > + * representing them in the view altogether. We still loop through
> > + * their counters so that we can assert that all values are zero.
> > + */
> > + bktype_tracked = pgstat_tracks_io_bktype(bktype);
>
> How about instead just doing Assert(pgstat_bktype_io_stats_valid(...))? That
> deduplicates the logic for the asserts, and avoids doing the full loop when
> assertions aren't enabled anyway?
>
I've done this and added a comment.
>
>
> > +-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
> > +-- and fsyncs.
> > +-- The second checkpoint ensures that stats from the first checkpoint have been
> > +-- reported and protects against any potential races amongst the table
> > +-- creation, a possible timing-triggered checkpoint, and the explicit
> > +-- checkpoint in the test.
>
> There's a comment about the subsequent checkpoints earlier in the file, and I
> think the comment is slightly more precise. Mybe just reference the earlier comment?
>
>
> > +-- Change the tablespace so that the table is rewritten directly, then SELECT
> > +-- from it to cause it to be read back into shared buffers.
> > +SET allow_in_place_tablespaces = true;
> > +CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
>
> Perhaps worth doing this in tablespace.sql, to avoid the additional
> checkpoints done as part of CREATE/DROP TABLESPACE?
>
> Or, at least combine this with the CHECKPOINTs above?
I see a checkpoint is requested when dropping the tablespace if not all
the files in it are deleted. It seems like if the DROP TABLE for the
permanent table is before the explicit checkpoints in the test, then the
DROP TABLESPACE will not cause an additional checkpoint. Is this what
you are suggesting? Dropping the temporary table should not have an
effect on this.
>
> > +-- Drop the table so we can drop the tablespace later.
> > +DROP TABLE test_io_shared;
> > +-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
> > +-- - eviction of local buffers in order to reuse them
> > +-- - reads of temporary table blocks into local buffers
> > +-- - writes of local buffers to permanent storage
> > +-- - extends of temporary tables
> > +-- Set temp_buffers to a low value so that we can trigger writes with fewer
> > +-- inserted tuples. Do so in a new session in case temporary tables have been
> > +-- accessed by previous tests in this session.
> > +\c
> > +SET temp_buffers TO '1MB';
>
> I'd set it to the actual minimum '100' (in pages). Perhaps that'd allow to
> make test_io_local a bit smaller?
I've done this.
>
> > +CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
> > +SELECT sum(extends) AS io_sum_local_extends_before
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> > +SELECT sum(evictions) AS io_sum_local_evictions_before
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> > +SELECT sum(writes) AS io_sum_local_writes_before
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> > +-- Insert tuples into the temporary table, generating extends in the stats.
> > +-- Insert enough values that we need to reuse and write out dirty local
> > +-- buffers, generating evictions and writes.
> > +INSERT INTO test_io_local SELECT generate_series(1, 8000) as id, repeat('a', 100);
> > +SELECT sum(reads) AS io_sum_local_reads_before
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
>
> Maybe add something like
>
> SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
>
> Better toast compression or such could easily make test_io_local smaller than
> it's today. Seeing that it's too small would make it easier to understand the
> failure.
Good idea. So, I used pg_table_size() because it seems like
pg_relation_size() does not include the toast relations. However, I'm
not sure this is a good idea, because pg_table_size() includes FSM and
visibility map. Should I write a query to get the toast relation name
and add pg_relation_size() of that relation and the main relation?
>
> > +SELECT sum(evictions) AS io_sum_local_evictions_after
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> > +SELECT sum(reads) AS io_sum_local_reads_after
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> > +SELECT sum(writes) AS io_sum_local_writes_after
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
> > +SELECT sum(extends) AS io_sum_local_extends_after
> > + FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
>
> This could just be one select with multiple columns?
>
> I think if you use something like \gset io_sum_local_after_ you can also avoid
> the need to repeat "io_sum_local_" so many times.
Thanks. I didn't realize. I've fixed this throughout the test file.
On Mon, Jan 16, 2023 at 4:42 PM Maciek Sakrejda <[email protected]> wrote:
> I missed a couple of versions, but I think the docs are clearer now.
> I'm torn on losing some of the detail, but overall I do think it's a
> good trade-off. Moving some details out to after the table does keep
> the bulk of the view documentation more readable, and the "inform
> database tuning" part is great. I really like the idea of a separate
> Interpreting Statistics section, but for now this works.
>
> >+ <literal>vacuum</literal>: I/O operations performed outside of shared
> >+ buffers while vacuuming and analyzing permanent relations.
>
> Why only permanent relations? Are temporary relations treated
> differently? I imagine if someone has a temp-table-heavy workload that
> requires regularly vacuuming and analyzing those relations, this point
> may be confusing without some additional explanation.
Ah, yes. This is a bit confusing. We don't use buffer access strategies
when operating on temp relations, so vacuuming them is counted in IO
Context normal. I've added this information to the docs but now that
definition is a bit long. Perhaps it should be a note? That seems like
it would draw too much attention to this detail, though...
- Melanie
Attachments:
[text/x-patch] v48-0002-pgstat-Count-IO-for-relations.patch (22.2K, ../../CAAKRu_bCU3rqyjvWYszSHMvQRRYtJjmzV49MJqsX6wY9ftoazA@mail.gmail.com/2-v48-0002-pgstat-Count-IO-for-relations.patch)
download | inline diff:
From 7a7a254c218df04f513cef4b7b2c38725d58a8a4 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Wed, 4 Jan 2023 17:20:50 -0500
Subject: [PATCH v48 2/4] pgstat: Count IO for relations
Count IOOps done on IOObjects in IOContexts by various BackendTypes
using the IO stats infrastructure introduced by a previous commit.
The primary concern of these statistics is IO operations on data blocks
during the course of normal database operations. IO operations done by,
for example, the archiver or syslogger are not counted in these
statistics. WAL IO, temporary file IO, and IO done directly though smgr*
functions (such as when building an index) are not yet counted but would
be useful future additions.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
src/backend/storage/buffer/bufmgr.c | 109 ++++++++++++++++++++++----
src/backend/storage/buffer/freelist.c | 58 ++++++++++----
src/backend/storage/buffer/localbuf.c | 13 ++-
src/backend/storage/smgr/md.c | 24 ++++++
src/include/storage/buf_internals.h | 8 +-
src/include/storage/bufmgr.h | 7 +-
6 files changed, 183 insertions(+), 36 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8075828e8a..484f422b72 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -481,8 +481,9 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr);
-static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
+ bool *foundPtr, IOContext *io_context);
+static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
+ IOContext io_context, IOObject io_object);
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -823,6 +824,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BufferDesc *bufHdr;
Block bufBlock;
bool found;
+ IOContext io_context;
+ IOObject io_object;
bool isExtend;
bool isLocalBuf = SmgrIsTemp(smgr);
@@ -855,7 +858,14 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isLocalBuf)
{
- bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found);
+ /*
+ * LocalBufferAlloc() will set the io_context to IOCONTEXT_NORMAL. We
+ * do not use a BufferAccessStrategy for I/O of temporary tables.
+ * However, in some cases, the "strategy" may not be NULL, so we can't
+ * rely on IOContextForStrategy() to set the right IOContext for us.
+ * This may happen in cases like CREATE TEMPORARY TABLE AS...
+ */
+ bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found, &io_context);
if (found)
pgBufferUsage.local_blks_hit++;
else if (isExtend)
@@ -871,7 +881,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, &found, &io_context);
if (found)
pgBufferUsage.shared_blks_hit++;
else if (isExtend)
@@ -986,7 +996,16 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
Assert(!(pg_atomic_read_u32(&bufHdr->state) & BM_VALID)); /* spinlock not needed */
- bufBlock = isLocalBuf ? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr);
+ if (isLocalBuf)
+ {
+ bufBlock = LocalBufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ bufBlock = BufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_RELATION;
+ }
if (isExtend)
{
@@ -995,6 +1014,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
/* don't set checksum for all-zero page */
smgrextend(smgr, forkNum, blockNum, (char *) bufBlock, false);
+ pgstat_count_io_op(IOOP_EXTEND, io_object, io_context);
+
/*
* NB: we're *not* doing a ScheduleBufferTagForWriteback here;
* although we're essentially performing a write. At least on linux
@@ -1020,6 +1041,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgrread(smgr, forkNum, blockNum, (char *) bufBlock);
+ pgstat_count_io_op(IOOP_READ, io_object, io_context);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -1113,14 +1136,19 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* *foundPtr is actually redundant with the buffer's BM_VALID flag, but
* we keep it for simplicity in ReadBuffer.
*
+ * io_context is passed as an output parameter to avoid calling
+ * IOContextForStrategy() when there is a shared buffers hit and no IO
+ * statistics need be captured.
+ *
* No locks are held either at entry or exit.
*/
static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
+ bool from_ring;
BufferTag newTag; /* identity of requested block */
uint32 newHash; /* hash value for newTag */
LWLock *newPartitionLock; /* buffer partition lock for it */
@@ -1172,8 +1200,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* If we get here, previous attempts to read the buffer must
- * have failed ... but we shall bravely try again.
+ * have failed ... but we shall bravely try again. Set
+ * io_context since we will in fact need to count an IO
+ * Operation.
*/
+ *io_context = IOContextForStrategy(strategy);
*foundPtr = false;
}
}
@@ -1187,6 +1218,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ *io_context = IOContextForStrategy(strategy);
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1200,7 +1233,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* Select a victim buffer. The buffer is returned with its header
* spinlock still held!
*/
- buf = StrategyGetBuffer(strategy, &buf_state);
+ buf = StrategyGetBuffer(strategy, &buf_state, &from_ring);
Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 0);
@@ -1254,7 +1287,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
if (XLogNeedsFlush(lsn) &&
- StrategyRejectBuffer(strategy, buf))
+ StrategyRejectBuffer(strategy, buf, from_ring))
{
/* Drop lock/pin and loop around for another buffer */
LWLockRelease(BufferDescriptorGetContentLock(buf));
@@ -1269,7 +1302,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgr->smgr_rlocator.locator.dbOid,
smgr->smgr_rlocator.locator.relNumber);
- FlushBuffer(buf, NULL);
+ FlushBuffer(buf, NULL, *io_context, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(buf));
ScheduleBufferTagForWriteback(&BackendWritebackContext,
@@ -1450,6 +1483,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
LWLockRelease(newPartitionLock);
+ if (oldFlags & BM_VALID)
+ {
+ /*
+ * When a BufferAccessStrategy is in use, blocks evicted from shared
+ * buffers are counted as IOOP_EVICT in the corresponding context
+ * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
+ * strategy in two cases: 1) while initially claiming buffers for the
+ * strategy ring 2) to replace an existing strategy ring buffer
+ * because it is pinned or in use and cannot be reused.
+ *
+ * Blocks evicted from buffers already in the strategy ring are
+ * counted as IOOP_REUSE in the corresponding strategy context.
+ *
+ * At this point, we can accurately count evictions and reuses,
+ * because we have successfully claimed the valid buffer. Previously,
+ * we may have been forced to release the buffer due to concurrent
+ * pinners or erroring out.
+ */
+ pgstat_count_io_op(from_ring ? IOOP_REUSE : IOOP_EVICT,
+ IOOBJECT_RELATION, *io_context);
+ }
+
/*
* Buffer contents are currently invalid. Try to obtain the right to
* start I/O. If StartBufferIO returns false, then someone else managed
@@ -2570,7 +2625,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
@@ -2820,7 +2875,7 @@ BufferGetTag(Buffer buffer, RelFileLocator *rlocator, ForkNumber *forknum,
* as the second parameter. If not, pass NULL.
*/
static void
-FlushBuffer(BufferDesc *buf, SMgrRelation reln)
+FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOContext io_context, IOObject io_object)
{
XLogRecPtr recptr;
ErrorContextCallback errcallback;
@@ -2912,6 +2967,26 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
bufToWrite,
false);
+ /*
+ * When a strategy is in use, only flushes of dirty buffers already in the
+ * strategy ring are counted as strategy writes (IOCONTEXT
+ * [BULKREAD|BULKWRITE|VACUUM] IOOP_WRITE) for the purpose of IO
+ * statistics tracking.
+ *
+ * If a shared buffer initially added to the ring must be flushed before
+ * being used, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE.
+ *
+ * If a shared buffer which was added to the ring later because the
+ * current strategy buffer is pinned or in use or because all strategy
+ * buffers were dirty and rejected (for BAS_BULKREAD operations only)
+ * requires flushing, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE
+ * (from_ring will be false).
+ *
+ * When a strategy is not in use, the write can only be a "regular" write
+ * of a dirty shared buffer (IOCONTEXT_NORMAL IOOP_WRITE).
+ */
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_RELATION, io_context);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -3554,6 +3629,8 @@ FlushRelationBuffers(Relation rel)
buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -3586,7 +3663,7 @@ FlushRelationBuffers(Relation rel)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, RelationGetSmgr(rel));
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3684,7 +3761,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, srelent->srel);
+ FlushBuffer(bufHdr, srelent->srel, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3894,7 +3971,7 @@ FlushDatabaseBuffers(Oid dbid)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3921,7 +3998,7 @@ FlushOneBuffer(Buffer buffer)
Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr)));
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOCONTEXT_NORMAL, IOOBJECT_RELATION);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7dec35801c..c690d5f15f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,7 @@
*/
#include "postgres.h"
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -81,12 +82,6 @@ typedef struct BufferAccessStrategyData
*/
int current;
- /*
- * True if the buffer just returned by StrategyGetBuffer had been in the
- * ring already.
- */
- bool current_was_in_ring;
-
/*
* Array of buffer numbers. InvalidBuffer (that is, zero) indicates we
* have not yet selected a buffer for this ring slot. For allocation
@@ -198,13 +193,15 @@ have_free_buffer(void)
* return the buffer with the buffer header spinlock still held.
*/
BufferDesc *
-StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
+StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring)
{
BufferDesc *buf;
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ *from_ring = false;
+
/*
* If given a strategy object, see whether it can select a buffer. We
* assume strategy objects don't need buffer_strategy_lock.
@@ -213,7 +210,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
{
buf = GetBufferFromRing(strategy, buf_state);
if (buf != NULL)
+ {
+ *from_ring = true;
return buf;
+ }
}
/*
@@ -602,7 +602,7 @@ FreeAccessStrategy(BufferAccessStrategy strategy)
/*
* GetBufferFromRing -- returns a buffer from the ring, or NULL if the
- * ring is empty.
+ * ring is empty / not usable.
*
* The bufhdr spin lock is held on the returned buffer.
*/
@@ -625,10 +625,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
*/
bufnum = strategy->buffers[strategy->current];
if (bufnum == InvalidBuffer)
- {
- strategy->current_was_in_ring = false;
return NULL;
- }
/*
* If the buffer is pinned we cannot use it under any circumstances.
@@ -644,7 +641,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) <= 1)
{
- strategy->current_was_in_ring = true;
*buf_state = local_buf_state;
return buf;
}
@@ -654,7 +650,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
* Tell caller to allocate a new buffer with the normal allocation
* strategy. He'll then replace this ring element via AddBufferToRing.
*/
- strategy->current_was_in_ring = false;
return NULL;
}
@@ -670,6 +665,39 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
strategy->buffers[strategy->current] = BufferDescriptorGetBuffer(buf);
}
+/*
+ * Utility function returning the IOContext of a given BufferAccessStrategy's
+ * strategy ring.
+ */
+IOContext
+IOContextForStrategy(BufferAccessStrategy strategy)
+{
+ if (!strategy)
+ return IOCONTEXT_NORMAL;
+
+ switch (strategy->btype)
+ {
+ case BAS_NORMAL:
+
+ /*
+ * Currently, GetAccessStrategy() returns NULL for
+ * BufferAccessStrategyType BAS_NORMAL, so this case is
+ * unreachable.
+ */
+ pg_unreachable();
+ return IOCONTEXT_NORMAL;
+ case BAS_BULKREAD:
+ return IOCONTEXT_BULKREAD;
+ case BAS_BULKWRITE:
+ return IOCONTEXT_BULKWRITE;
+ case BAS_VACUUM:
+ return IOCONTEXT_VACUUM;
+ }
+
+ elog(ERROR, "unrecognized BufferAccessStrategyType: %d", strategy->btype);
+ pg_unreachable();
+}
+
/*
* StrategyRejectBuffer -- consider rejecting a dirty buffer
*
@@ -682,14 +710,14 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
* if this buffer should be written and re-used.
*/
bool
-StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
+StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring)
{
/* We only do this in bulkread mode */
if (strategy->btype != BAS_BULKREAD)
return false;
/* Don't muck with behavior of normal buffer-replacement strategy */
- if (!strategy->current_was_in_ring ||
+ if (!from_ring ||
strategy->buffers[strategy->current] != BufferDescriptorGetBuffer(buf))
return false;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8372acc383..2108bbe7d8 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "catalog/catalog.h"
#include "executor/instrument.h"
+#include "pgstat.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "utils/guc_hooks.h"
@@ -107,7 +108,7 @@ PrefetchLocalBuffer(SMgrRelation smgr, ForkNumber forkNum,
*/
BufferDesc *
LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
BufferTag newTag; /* identity of requested block */
LocalBufferLookupEnt *hresult;
@@ -127,6 +128,14 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
+ /*
+ * IO Operations on local buffers are only done in IOCONTEXT_NORMAL. Set
+ * io_context here (instead of after a buffer hit would have returned) for
+ * convenience since we don't have to worry about the overhead of calling
+ * IOContextForStrategy().
+ */
+ *io_context = IOCONTEXT_NORMAL;
+
if (hresult)
{
b = hresult->id;
@@ -230,6 +239,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
buf_state &= ~BM_DIRTY;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_WRITE, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
pgBufferUsage.local_blks_written++;
}
@@ -256,6 +266,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
ClearBufferTag(&bufHdr->tag);
buf_state &= ~(BM_VALID | BM_TAG_VALID);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOP_EVICT, IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL);
}
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 60c9905eff..58ae2af55a 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -983,6 +983,15 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
{
MdfdVec *v = &reln->md_seg_fds[forknum][segno - 1];
+ /*
+ * fsyncs done through mdimmedsync() should be tracked in a separate
+ * IOContext than those done through mdsyncfiletag() to differentiate
+ * between unavoidable client backend fsyncs (e.g. those done during
+ * index build) and those which ideally would have been done by the
+ * checkpointer. Since other IO operations bypassing the buffer
+ * manager could also be tracked in such an IOContext, wait until
+ * these are also tracked to track immediate fsyncs.
+ */
if (FileSync(v->mdfd_vfd, WAIT_EVENT_DATA_FILE_IMMEDIATE_SYNC) < 0)
ereport(data_sync_elevel(ERROR),
(errcode_for_file_access(),
@@ -1021,6 +1030,19 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
{
+ /*
+ * We have no way of knowing if the current IOContext is
+ * IOCONTEXT_NORMAL or IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] at this
+ * point, so count the fsync as being in the IOCONTEXT_NORMAL
+ * IOContext. This is probably okay, because the number of backend
+ * fsyncs doesn't say anything about the efficacy of the
+ * BufferAccessStrategy. And counting both fsyncs done in
+ * IOCONTEXT_NORMAL and IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] under
+ * IOCONTEXT_NORMAL is likely clearer when investigating the number of
+ * backend fsyncs.
+ */
+ pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+
ereport(DEBUG1,
(errmsg_internal("could not forward fsync request because request queue is full")));
@@ -1410,6 +1432,8 @@ mdsyncfiletag(const FileTag *ftag, char *path)
if (need_to_close)
FileClose(file);
+ pgstat_count_io_op(IOOP_FSYNC, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+
errno = save_errno;
return result;
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index ed8aa2519c..0b44814740 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -15,6 +15,7 @@
#ifndef BUFMGR_INTERNALS_H
#define BUFMGR_INTERNALS_H
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
@@ -391,11 +392,12 @@ extern void IssuePendingWritebacks(WritebackContext *context);
extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *tag);
/* freelist.c */
+extern IOContext IOContextForStrategy(BufferAccessStrategy bas);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
- uint32 *buf_state);
+ uint32 *buf_state, bool *from_ring);
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
- BufferDesc *buf);
+ BufferDesc *buf, bool from_ring);
extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
extern void StrategyNotifyBgWriter(int bgwprocno);
@@ -417,7 +419,7 @@ extern PrefetchBufferResult PrefetchLocalBuffer(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum);
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
- BlockNumber blockNum, bool *foundPtr);
+ BlockNumber blockNum, bool *foundPtr, IOContext *io_context);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 33eadbc129..b8a18b8081 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -23,7 +23,12 @@
typedef void *Block;
-/* Possible arguments for GetAccessStrategy() */
+/*
+ * Possible arguments for GetAccessStrategy().
+ *
+ * If adding a new BufferAccessStrategyType, also add a new IOContext so
+ * IO statistics using this strategy are tracked.
+ */
typedef enum BufferAccessStrategyType
{
BAS_NORMAL, /* Normal random access */
--
2.34.1
[text/x-patch] v48-0003-Add-system-view-tracking-IO-ops-per-backend-type.patch (32.6K, ../../CAAKRu_bCU3rqyjvWYszSHMvQRRYtJjmzV49MJqsX6wY9ftoazA@mail.gmail.com/3-v48-0003-Add-system-view-tracking-IO-ops-per-backend-type.patch)
download | inline diff:
From 879cce732348d599445802f00e8650ee521fb239 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 9 Jan 2023 14:42:25 -0500
Subject: [PATCH v48 3/4] Add system view tracking IO ops per backend type
Add pg_stat_io, a system view which tracks the number of IOOps
(evictions, reuses, reads, writes, extensions, and fsyncs) done on each
IOObject (relation, temp relation) in each IOContext ("normal" and those
using a BufferAccessStrategy) by each type of backend (e.g. client
backend, checkpointer).
Some BackendTypes do not accumulate IO operations statistics and will
not be included in the view.
Some IOContexts are not used by some BackendTypes and will not be in the
view. For example, checkpointer does not use a BufferAccessStrategy
(currently), so there will be no rows for BufferAccessStrategy
IOContexts for checkpointer.
Some IOObjects are never operated on in some IOContexts or by some
BackendTypes. These rows are omitted from the view. For example,
checkpointer will never operate on IOOBJECT_TEMP_RELATION data, so those
rows are omitted.
Some IOOps are invalid in combination with certain IOContexts and
certain IOObjects. Those cells will be NULL in the view to distinguish
between 0 observed IOOps of that type and an invalid combination. For
example, temporary tables are not fsynced so cells for all BackendTypes
for IOOBJECT_TEMP_RELATION and IOOP_FSYNC will be NULL.
Some BackendTypes never perform certain IOOps. Those cells will also be
NULL in the view. For example, bgwriter should not perform reads.
View stats are populated with statistics incremented when a backend
performs an IO Operation and maintained by the cumulative statistics
subsystem.
Each row of the view shows stats for a particular BackendType, IOObject,
IOContext combination (e.g. a client backend's operations on permanent
relations in shared buffers) and each column in the view is the total
number of IO Operations done (e.g. writes). So a cell in the view would
be, for example, the number of blocks of relation data written from
shared buffers by client backends since the last stats reset.
In anticipation of tracking WAL IO and non-block-oriented IO (such as
temporary file IO), the "op_bytes" column specifies the unit of the
"reads", "writes", and "extends" columns for a given row.
Note that some of the cells in the view are redundant with fields in
pg_stat_bgwriter (e.g. buffers_backend), however these have been kept in
pg_stat_bgwriter for backwards compatibility. Deriving the redundant
pg_stat_bgwriter stats from the IO operations stats structures was also
problematic due to the separate reset targets for 'bgwriter' and 'io'.
Suggested by Andres Freund
Catalog version should be bumped.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
contrib/amcheck/expected/check_heap.out | 34 ++++
contrib/amcheck/sql/check_heap.sql | 27 +++
src/backend/catalog/system_views.sql | 15 ++
src/backend/utils/adt/pgstatfuncs.c | 140 ++++++++++++++
src/include/catalog/pg_proc.dat | 9 +
src/test/regress/expected/rules.out | 12 ++
src/test/regress/expected/stats.out | 234 ++++++++++++++++++++++++
src/test/regress/sql/stats.sql | 148 +++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 620 insertions(+)
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index c010361025..e4785141a6 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -66,6 +66,22 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -88,6 +104,23 @@ SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock :=
-------+--------+--------+-----
(0 rows)
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+ ?column?
+----------
+ t
+(1 row)
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
SET ROLE regress_heaptest_role;
@@ -195,6 +228,7 @@ ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 298de6886a..6794ca4eb0 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -20,11 +20,29 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
+
-- Add some data so subsequent tests are not entirely trivial
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
+
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -32,6 +50,14 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');
SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
@@ -110,6 +136,7 @@ SELECT * FROM verify_heapam('test_foreign_table',
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d2a8c82900..f875742068 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1116,6 +1116,21 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+CREATE VIEW pg_stat_io AS
+SELECT
+ b.backend_type,
+ b.io_context,
+ b.io_object,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+FROM pg_stat_get_io() b;
+
CREATE VIEW pg_stat_wal AS
SELECT
w.wal_records,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 6df9f06a20..284bb2a698 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1234,6 +1234,146 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+* When adding a new column to the pg_stat_io view, add a new enum value
+* here above IO_NUM_COLUMNS.
+*/
+typedef enum io_stat_col
+{
+ IO_COL_BACKEND_TYPE,
+ IO_COL_IO_CONTEXT,
+ IO_COL_IO_OBJECT,
+ IO_COL_READS,
+ IO_COL_WRITES,
+ IO_COL_EXTENDS,
+ IO_COL_CONVERSION,
+ IO_COL_EVICTIONS,
+ IO_COL_REUSES,
+ IO_COL_FSYNCS,
+ IO_COL_RESET_TIME,
+ IO_NUM_COLUMNS,
+} io_stat_col;
+
+/*
+ * When adding a new IOOp, add a new io_stat_col and add a case to this
+ * function returning the corresponding io_stat_col.
+ */
+static io_stat_col
+pgstat_get_io_op_index(IOOp io_op)
+{
+ switch (io_op)
+ {
+ case IOOP_EVICT:
+ return IO_COL_EVICTIONS;
+ case IOOP_READ:
+ return IO_COL_READS;
+ case IOOP_REUSE:
+ return IO_COL_REUSES;
+ case IOOP_WRITE:
+ return IO_COL_WRITES;
+ case IOOP_EXTEND:
+ return IO_COL_EXTENDS;
+ case IOOP_FSYNC:
+ return IO_COL_FSYNCS;
+ }
+
+ elog(ERROR, "unrecognized IOOp value: %d", io_op);
+ pg_unreachable();
+}
+
+Datum
+pg_stat_get_io(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo;
+ PgStat_IO *backends_io_stats;
+ Datum reset_time;
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ backends_io_stats = pgstat_fetch_stat_io();
+
+ reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
+
+ for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ {
+ Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
+ PgStat_BktypeIO *bktype_stats = &backends_io_stats->stats[bktype];
+
+ /*
+ * In Assert builds, we can afford an extra loop through all of the
+ * counters checking that only expected stats are non-zero, since it
+ * keeps the non-Assert code cleaner.
+ */
+ Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype));
+
+ /*
+ * For those BackendTypes without IO Operation stats, skip
+ * representing them in the view altogether.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ continue;
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ const char *context_name = pgstat_get_io_context_name(io_context);
+
+ for (IOObject io_obj = IOOBJECT_FIRST;
+ io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ {
+ const char *obj_name = pgstat_get_io_object_name(io_obj);
+
+ Datum values[IO_NUM_COLUMNS] = {0};
+ bool nulls[IO_NUM_COLUMNS] = {0};
+
+ /*
+ * Some combinations of IOContext, IOObject, and BackendType
+ * are not valid for any type of IOOp. In such cases, omit the
+ * entire row from the view.
+ */
+ if (!pgstat_tracks_io_object(bktype, io_context, io_obj))
+ continue;
+
+ values[IO_COL_BACKEND_TYPE] = bktype_desc;
+ values[IO_COL_IO_CONTEXT] = CStringGetTextDatum(context_name);
+ values[IO_COL_IO_OBJECT] = CStringGetTextDatum(obj_name);
+ values[IO_COL_RESET_TIME] = TimestampTzGetDatum(reset_time);
+
+ /*
+ * Hard-code this to the value of BLCKSZ for now. Future
+ * values could include XLOG_BLCKSZ, once WAL IO is tracked,
+ * and constant multipliers, once non-block-oriented IO (e.g.
+ * temporary file IO) is tracked.
+ */
+ values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ int col_idx = pgstat_get_io_op_index(io_op);
+
+ /*
+ * Some combinations of BackendType and IOOp, of IOContext
+ * and IOOp, and of IOObject and IOOp are not tracked. Set
+ * these cells in the view NULL.
+ */
+ nulls[col_idx] = !pgstat_tracks_io_op(bktype, io_context, io_obj, io_op);
+
+ if (nulls[col_idx])
+ continue;
+
+ values[col_idx] =
+ Int64GetDatum(bktype_stats->data[io_context][io_obj][io_op]);
+ }
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ }
+ }
+
+ return (Datum) 0;
+}
+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3810de7b22..57a889cf49 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5690,6 +5690,15 @@
proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
+{ oid => '8459', descr => 'statistics: per backend type IO statistics',
+ proname => 'pg_stat_get_io', provolatile => 'v',
+ prorows => '30', proretset => 't',
+ proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,text,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{backend_type,io_context,io_object,reads,writes,extends,op_bytes,evictions,reuses,fsyncs,stats_reset}',
+ prosrc => 'pg_stat_get_io' },
+
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index a969ae63eb..dd5ddffc4d 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1876,6 +1876,18 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_enc AS encrypted
FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
WHERE (s.client_port IS NOT NULL);
+pg_stat_io| SELECT b.backend_type,
+ b.io_context,
+ b.io_object,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+ FROM pg_stat_get_io() b(backend_type, io_context, io_object, reads, writes, extends, op_bytes, evictions, reuses, fsyncs, stats_reset);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 1d84407a03..3bd4e66fa8 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1126,4 +1126,238 @@ SELECT pg_stat_get_subscription_stats(NULL);
(1 row)
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+ ?column?
+----------
+ t
+(1 row)
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+ count
+-------
+ 100
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_table_size('test_io_local') / current_setting('block_size')::int8 > 100;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+ count
+-------
+ 5000
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+ ?column? | ?column? | ?column? | ?column?
+----------+----------+----------+----------
+ t | t | t | t
+(1 row)
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
+
+RESET wal_skip_threshold;
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+ pg_stat_have_stats
+--------------------
+ t
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+ ?column?
+----------
+ t
+(1 row)
+
-- End of Stats Test
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index b4d6753c71..163ed38faf 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -536,4 +536,152 @@ SELECT pg_stat_get_replication_slot(NULL);
SELECT pg_stat_get_subscription_stats(NULL);
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_table_size('test_io_local') / current_setting('block_size')::int8 > 100;
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+SELECT pg_stat_force_next_flush();
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+RESET wal_skip_threshold;
+
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+
-- End of Stats Test
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1be6e07980..a399e0a5e4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3377,6 +3377,7 @@ intset_internal_node
intset_leaf_node
intset_node
intvKEY
+io_stat_col
itemIdCompact
itemIdCompactData
iterator
--
2.34.1
[text/x-patch] v48-0004-pg_stat_io-documentation.patch (13.5K, ../../CAAKRu_bCU3rqyjvWYszSHMvQRRYtJjmzV49MJqsX6wY9ftoazA@mail.gmail.com/4-v48-0004-pg_stat_io-documentation.patch)
download | inline diff:
From 21d63527d9e62afb5649bcbf162c3be860408f66 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 9 Jan 2023 14:42:53 -0500
Subject: [PATCH v48 4/4] pg_stat_io documentation
Author: Melanie Plageman <[email protected]>
Author: Samay Sharma <[email protected]>
Reviewed-by: Maciek Sakrejda <[email protected]>
Reviewed-by: Lukas Fittl <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 321 +++++++++++++++++++++++++++++++++--
1 file changed, 307 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 8d51ca3773..d0ca41e204 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -469,6 +469,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
+ <entry>
+ One row for each combination of backend type, context, and target object
+ containing cluster-wide I/O statistics.
+ See <link linkend="monitoring-pg-stat-io-view">
+ <structname>pg_stat_io</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_replication_slots</structname><indexterm><primary>pg_stat_replication_slots</primary></indexterm></entry>
<entry>One row per replication slot, showing statistics about the
@@ -665,20 +675,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</para>
<para>
- The <structname>pg_statio_</structname> views are primarily useful to
- determine the effectiveness of the buffer cache. When the number
- of actual disk reads is much smaller than the number of buffer
- hits, then the cache is satisfying most read requests without
- invoking a kernel call. However, these statistics do not give the
- entire story: due to the way in which <productname>PostgreSQL</productname>
- handles disk I/O, data that is not in the
- <productname>PostgreSQL</productname> buffer cache might still reside in the
- kernel's I/O cache, and might therefore still be fetched without
- requiring a physical read. Users interested in obtaining more
- detailed information on <productname>PostgreSQL</productname> I/O behavior are
- advised to use the <productname>PostgreSQL</productname> statistics views
- in combination with operating system utilities that allow insight
- into the kernel's handling of I/O.
+ The <structname>pg_stat_io</structname> and
+ <structname>pg_statio_</structname> set of views are useful for determining
+ the effectiveness of the buffer cache. They can be used to calculate a cache
+ hit ratio. Note that while <productname>PostgreSQL</productname>'s I/O
+ statistics capture most instances in which the kernel was invoked in order
+ to perform I/O, they do not differentiate between data which had to be
+ fetched from disk and that which already resided in the kernel page cache.
+ Users are advised to use the <productname>PostgreSQL</productname>
+ statistics views in combination with operating system utilities for a more
+ complete picture of their database's I/O performance.
</para>
</sect2>
@@ -3643,6 +3649,293 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_archived_wal</structfield> have also been successfully
archived.
</para>
+ </sect2>
+
+ <sect2 id="monitoring-pg-stat-io-view">
+ <title><structname>pg_stat_io</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_io</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_io</structname> view will contain one row for each
+ combination of backend type, I/O context, and target I/O object showing
+ cluster-wide I/O statistics. Combinations which do not make sense are
+ omitted.
+ </para>
+
+ <para>
+ Currently, I/O on relations (e.g. tables, indexes) is tracked. However,
+ relation I/O which bypasses shared buffers (e.g. when moving a table from one
+ tablespace to another) is currently not tracked.
+ </para>
+
+ <table id="pg-stat-io-view" xreflabel="pg_stat_io">
+ <title><structname>pg_stat_io</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>backend_type</structfield> <type>text</type>
+ </para>
+ <para>
+ Type of backend (e.g. background worker, autovacuum worker). See <link
+ linkend="monitoring-pg-stat-activity-view">
+ <structname>pg_stat_activity</structname></link> for more information
+ on <varname>backend_type</varname>s. Some
+ <varname>backend_type</varname>s do not accumulate I/O operation
+ statistics and will not be included in the view.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_context</structfield> <type>text</type>
+ </para>
+ <para>
+ The context of an I/O operation. Possible values are:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>normal</literal>: The default or standard
+ <varname>io_context</varname> for a type of I/O operation. For
+ example, by default, relation data is read into and written out from
+ shared buffers. Thus, reads and writes of relation data to and from
+ shared buffers are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>vacuum</literal>: I/O operations performed outside of shared
+ buffers while vacuuming and analyzing permanent relations. Temporary
+ table vacuums use the same local buffer pool as other temporary table
+ IO operations and are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkread</literal>: Certain large read I/O operations
+ done outside of shared buffers, for example, a sequential scan of a
+ large table.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkwrite</literal>: Certain large write I/O operations
+ done outside of shared buffers, such as <command>COPY</command>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_object</structfield> <type>text</type>
+ </para>
+ <para>
+ Target object of an I/O operation. Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>relation</literal>: Permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>temp relation</literal>: Temporary relations.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reads</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of read operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>writes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of write operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>extends</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of relation extend operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>op_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of bytes per unit of I/O read, written, or extended.
+ </para>
+ <para>
+ Relation data reads, writes, and extends are done in
+ <varname>block_size</varname> units, derived from the build-time
+ parameter <symbol>BLCKSZ</symbol>, which is <literal>8192</literal> by
+ default.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a block has been written out from a shared or local
+ buffer in order to make it available for another use.
+ </para>
+ <para>
+ In <varname>io_context</varname> <literal>normal</literal>, this counts
+ the number of times a block was evicted from a buffer and replaced with
+ another block. In <varname>io_context</varname>s
+ <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
+ <literal>vacuum</literal>, this counts the number of times a block was
+ evicted from shared buffers in order to add the shared buffer to a
+ separate, size-limited ring buffer for use in a bulk I/O operation.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reuses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of times an existing buffer in a size-limited ring buffer
+ outside of shared buffers was reused as part of an I/O operation in the
+ <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
+ <literal>vacuum</literal> <varname>io_context</varname>s.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>fsyncs</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of <literal>fsync</literal> calls. These are only tracked in
+ <varname>io_context</varname> <literal>normal</literal>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset.
+ </para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Some backend types never perform I/O operations in some I/O contexts and/or
+ on some I/O objects. These rows are omitted from the view. For example, the
+ checkpointer does not checkpoint temporary tables, so there will be no rows
+ for <varname>backend_type</varname> <literal>checkpointer</literal> and
+ <varname>io_object</varname> <literal>temp relation</literal>.
+ </para>
+
+ <para>
+ In addition, some I/O operations will never be performed either by certain
+ backend types or in certain I/O contexts or on certain I/O objects. These
+ cells will be NULL. For example, temporary tables are not
+ <literal>fsync</literal>ed, so <varname>fsyncs</varname> will be NULL for
+ <varname>io_object</varname> <literal>temp relation</literal>. Also, the
+ background writer does not perform reads, so <varname>reads</varname> will
+ be NULL in rows for <varname>backend_type</varname> <literal>background
+ writer</literal>.
+ </para>
+
+ <para>
+ <structname>pg_stat_io</structname> can be used to inform database tuning.
+ For example:
+ <itemizedlist>
+ <listitem>
+ <para>
+ A high <varname>evictions</varname> count can indicate that shared
+ buffers should be increased.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Client backends rely on the checkpointer to ensure data is persisted to
+ permanent storage. Large numbers of <varname>fsyncs</varname> by
+ <literal>client backend</literal>s could indicate a misconfiguration of
+ shared buffers or of the checkpointer. More information on configuring
+ the checkpointer can be found in <xref linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Normally, client backends should be able to rely on auxiliary processes
+ like the checkpointer and the background writer to write out dirty data
+ as much as possible. Large numbers of writes by client backends could
+ indicate a misconfiguration of shared buffers or of the checkpointer.
+ More information on configuring the checkpointer can be found in <xref
+ linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
</sect2>
--
2.34.1
[text/x-patch] v48-0001-pgstat-Infrastructure-to-track-IO-operations.patch (28.6K, ../../CAAKRu_bCU3rqyjvWYszSHMvQRRYtJjmzV49MJqsX6wY9ftoazA@mail.gmail.com/5-v48-0001-pgstat-Infrastructure-to-track-IO-operations.patch)
download | inline diff:
From f77f8e0eeb4377cbffd2d52c9455f05e41468bec Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Wed, 4 Jan 2023 17:20:41 -0500
Subject: [PATCH v48 1/4] pgstat: Infrastructure to track IO operations
Introduce "IOOp", an IO operation done by a backend, "IOObject", the
target object of the IO, and "IOContext", the context or location of the
IO operations on that object. For example, the checkpointer may write a
shared buffer out. This would be considered an IOOP_WRITE IOOp on an
IOOBJECT_RELATION IOObject in the IOCONTEXT_NORMAL IOContext by
BackendType B_CHECKPOINTER.
Each IOOp (evict, extend, fsync, read, reuse, and write) can be counted
per IOObject (relation, temp relation) per IOContext (normal, bulkread,
bulkwrite, or vacuum) through a call to pgstat_count_io_op().
Note that this commit introduces the infrastructure to count IO
Operation statistics. A subsequent commit will add calls to
pgstat_count_io_op() in the appropriate locations.
IOContext IOCONTEXT_NORMAL concerns operations on local and shared
buffers, while IOCONTEXT_BULKREAD, IOCONTEXT_BULKWRITE, and
IOCONTEXT_VACUUM IOContexts concern IO operations on buffers as part of
a BufferAccessStrategy.
IOObject IOOBJECT_TEMP_RELATION concerns IO Operations on buffers
containing temporary table data, while IOObject IOOBJECT_RELATION
concerns IO Operations on buffers containing permanent relation data.
Stats on IOOps on all IOObjects in all IOContexts for a given backend
are first counted in a backend's local memory and then flushed to shared
memory and accumulated with those from all other backends, exited and
live.
Some BackendTypes will not flush their pending statistics at regular
intervals and explicitly call pgstat_flush_io_ops() during the course of
normal operations to flush their backend-local IO operation statistics
to shared memory in a timely manner.
Because not all BackendType, IOOp, IOObject, IOContext combinations are
valid, the validity of the stats is checked before flushing pending
stats and before reading in the existing stats file to shared memory.
The aggregated stats in shared memory could be extended in the future
with per-backend stats -- useful for per connection IO statistics and
monitoring.
PGSTAT_FILE_FORMAT_ID should be bumped with this commit.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Reviewed-by: Kyotaro Horiguchi <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 2 +
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 26 ++
src/backend/utils/activity/pgstat_bgwriter.c | 7 +-
.../utils/activity/pgstat_checkpointer.c | 7 +-
src/backend/utils/activity/pgstat_io.c | 385 ++++++++++++++++++
src/backend/utils/activity/pgstat_relation.c | 15 +-
src/backend/utils/activity/pgstat_shmem.c | 4 +
src/backend/utils/activity/pgstat_wal.c | 4 +-
src/backend/utils/adt/pgstatfuncs.c | 11 +-
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 68 ++++
src/include/utils/pgstat_internal.h | 30 ++
src/tools/pgindent/typedefs.list | 6 +
15 files changed, 562 insertions(+), 7 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_io.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 358d2ff90f..8d51ca3773 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5418,6 +5418,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
the <structname>pg_stat_bgwriter</structname>
view, <literal>archiver</literal> to reset all the counters shown in
the <structname>pg_stat_archiver</structname> view,
+ <literal>io</literal> to reset all the counters shown in the
+ <structname>pg_stat_io</structname> view,
<literal>wal</literal> to reset all the counters shown in the
<structname>pg_stat_wal</structname> view or
<literal>recovery_prefetch</literal> to reset all the counters shown
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index a80eda3cf4..7d7482dde0 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -22,6 +22,7 @@ OBJS = \
pgstat_checkpointer.o \
pgstat_database.o \
pgstat_function.o \
+ pgstat_io.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index a2b872c24b..518ee3f798 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -9,6 +9,7 @@ backend_sources += files(
'pgstat_checkpointer.c',
'pgstat_database.c',
'pgstat_function.c',
+ 'pgstat_io.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 0fa5370bcd..60fc4e761f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -72,6 +72,7 @@
* - pgstat_checkpointer.c
* - pgstat_database.c
* - pgstat_function.c
+ * - pgstat_io.c
* - pgstat_relation.c
* - pgstat_replslot.c
* - pgstat_slru.c
@@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
+ [PGSTAT_KIND_IO] = {
+ .name = "io",
+
+ .fixed_amount = true,
+
+ .reset_all_cb = pgstat_io_reset_all_cb,
+ .snapshot_cb = pgstat_io_snapshot_cb,
+ },
+
[PGSTAT_KIND_SLRU] = {
.name = "slru",
@@ -582,6 +592,7 @@ pgstat_report_stat(bool force)
/* Don't expend a clock check if nothing to do */
if (dlist_is_empty(&pgStatPending) &&
+ !have_iostats &&
!have_slrustats &&
!pgstat_have_pending_wal())
{
@@ -628,6 +639,9 @@ pgstat_report_stat(bool force)
/* flush database / relation / function / ... stats */
partial_flush |= pgstat_flush_pending_entries(nowait);
+ /* flush IO stats */
+ partial_flush |= pgstat_flush_io(nowait);
+
/* flush wal stats */
partial_flush |= pgstat_flush_wal(nowait);
@@ -1322,6 +1336,12 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
+ /*
+ * Write IO stats struct
+ */
+ pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
+ write_chunk_s(fpout, &pgStatLocal.snapshot.io);
+
/*
* Write SLRU stats struct
*/
@@ -1496,6 +1516,12 @@ pgstat_read_statsfile(void)
if (!read_chunk_s(fpin, &shmem->checkpointer.stats))
goto error;
+ /*
+ * Read IO stats struct
+ */
+ if (!read_chunk_s(fpin, &shmem->io.stats))
+ goto error;
+
/*
* Read SLRU stats struct
*/
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 9247f2dda2..92be384b0d 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -24,7 +24,7 @@ PgStat_BgWriterStats PendingBgWriterStats = {0};
/*
- * Report bgwriter statistics
+ * Report bgwriter and IO statistics
*/
void
pgstat_report_bgwriter(void)
@@ -56,6 +56,11 @@ pgstat_report_bgwriter(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 3e9ab45103..26dec112f6 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -24,7 +24,7 @@ PgStat_CheckpointerStats PendingCheckpointerStats = {0};
/*
- * Report checkpointer statistics
+ * Report checkpointer and IO statistics
*/
void
pgstat_report_checkpointer(void)
@@ -62,6 +62,11 @@ pgstat_report_checkpointer(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingCheckpointerStats, 0, sizeof(PendingCheckpointerStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
new file mode 100644
index 0000000000..9859db0581
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -0,0 +1,385 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_io.c
+ * Implementation of IO statistics.
+ *
+ * This file contains the implementation of IO statistics. It is kept separate
+ * from pgstat.c to enforce the line between the statistics access / storage
+ * implementation and the details about individual types of statistics.
+ *
+ * Copyright (c) 2021-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_io.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "utils/pgstat_internal.h"
+
+
+static PgStat_BktypeIO PendingIOStats;
+bool have_iostats = false;
+
+/*
+ * Check that stats have not been counted for any combination of IOContext,
+ * IOObject, and IOOp which are not tracked for the passed-in BackendType. The
+ * passed-in PgStat_BktypeIO must contain stats from the BackendType specified
+ * by the second parameter. Caller is responsible for locking the passed-in
+ * PgStat_BktypeIO, if needed.
+ */
+bool
+pgstat_bktype_io_stats_valid(PgStat_BktypeIO *backend_io,
+ BackendType bktype)
+{
+ bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ /*
+ * Don't bother trying to skip to the next loop iteration if
+ * pgstat_tracks_io_object() would return false here. We still
+ * need to validate that each counter is zero anyway.
+ */
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ /* No stats, so nothing to validate */
+ if (backend_io->data[io_context][io_object][io_op] == 0)
+ continue;
+
+ /* There are stats and there shouldn't be */
+ if (!bktype_tracked ||
+ !pgstat_tracks_io_op(bktype, io_context, io_object, io_op))
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
+void
+pgstat_count_io_op(IOOp io_op, IOObject io_object, IOContext io_context)
+{
+ Assert(io_context < IOCONTEXT_NUM_TYPES);
+ Assert(io_object < IOOBJECT_NUM_TYPES);
+ Assert(io_op < IOOP_NUM_TYPES);
+ Assert(pgstat_tracks_io_op(MyBackendType, io_context, io_object, io_op));
+
+ PendingIOStats.data[io_context][io_object][io_op]++;
+
+ have_iostats = true;
+}
+
+PgStat_IO *
+pgstat_fetch_stat_io(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_IO);
+
+ return &pgStatLocal.snapshot.io;
+}
+
+/*
+ * Flush out locally pending IO statistics
+ *
+ * If no stats have been recorded, this function returns false.
+ *
+ * If nowait is true, this function returns true if the lock could not be
+ * acquired. Otherwise, return false.
+ */
+bool
+pgstat_flush_io(bool nowait)
+{
+ LWLock *bktype_lock;
+ PgStat_BktypeIO *bktype_shstats;
+
+ if (!have_iostats)
+ return false;
+
+ bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
+ bktype_shstats =
+ &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+
+ if (!nowait)
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
+ return true;
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOOp io_op = IOOP_FIRST;
+ io_op < IOOP_NUM_TYPES; io_op++)
+ bktype_shstats->data[io_context][io_object][io_op] +=
+ PendingIOStats.data[io_context][io_object][io_op];
+
+ Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
+
+ LWLockRelease(bktype_lock);
+
+ memset(&PendingIOStats, 0, sizeof(PendingIOStats));
+
+ have_iostats = false;
+
+ return false;
+}
+
+const char *
+pgstat_get_io_context_name(IOContext io_context)
+{
+ switch (io_context)
+ {
+ case IOCONTEXT_BULKREAD:
+ return "bulkread";
+ case IOCONTEXT_BULKWRITE:
+ return "bulkwrite";
+ case IOCONTEXT_NORMAL:
+ return "normal";
+ case IOCONTEXT_VACUUM:
+ return "vacuum";
+ }
+
+ elog(ERROR, "unrecognized IOContext value: %d", io_context);
+ pg_unreachable();
+}
+
+const char *
+pgstat_get_io_object_name(IOObject io_object)
+{
+ switch (io_object)
+ {
+ case IOOBJECT_RELATION:
+ return "relation";
+ case IOOBJECT_TEMP_RELATION:
+ return "temp relation";
+ }
+
+ elog(ERROR, "unrecognized IOObject value: %d", io_object);
+ pg_unreachable();
+}
+
+void
+pgstat_io_reset_all_cb(TimestampTz ts)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+
+ memset(bktype_shstats, 0, sizeof(*bktype_shstats));
+ LWLockRelease(bktype_lock);
+ }
+}
+
+void
+pgstat_io_snapshot_cb(void)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_SHARED);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.snapshot.io.stat_reset_timestamp =
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+
+ /* using struct assignment due to better type safety */
+ *bktype_snap = *bktype_shstats;
+ LWLockRelease(bktype_lock);
+ }
+}
+
+/*
+* IO statistics are not collected for all BackendTypes.
+*
+* The following BackendTypes do not participate in the cumulative stats
+* subsystem or do not perform IO on which we currently track:
+* - Syslogger because it is not connected to shared memory
+* - Archiver because most relevant archiving IO is delegated to a
+* specialized command or module
+* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
+*
+* Function returns true if BackendType participates in the cumulative stats
+* subsystem for IO and false if it does not.
+*
+* When adding a new BackendType, also consider adding relevant restrictions to
+* pgstat_tracks_io_object() and pgstat_tracks_io_op().
+*/
+bool
+pgstat_tracks_io_bktype(BackendType bktype)
+{
+ /*
+ * List every type so that new backend types trigger a warning about
+ * needing to adjust this switch.
+ */
+ switch (bktype)
+ {
+ case B_INVALID:
+ case B_ARCHIVER:
+ case B_LOGGER:
+ case B_WAL_RECEIVER:
+ case B_WAL_WRITER:
+ return false;
+
+ case B_AUTOVAC_LAUNCHER:
+ case B_AUTOVAC_WORKER:
+ case B_BACKEND:
+ case B_BG_WORKER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_STANDALONE_BACKEND:
+ case B_STARTUP:
+ case B_WAL_SENDER:
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Some BackendTypes do not perform IO in certain IOContexts. Some IOObjects
+ * are never operated on in some IOContexts. Check that the given BackendType
+ * is expected to do IO in the given IOContext and that the given IOObject is
+ * expected to be operated on in the given IOContext.
+ */
+bool
+pgstat_tracks_io_object(BackendType bktype, IOContext io_context,
+ IOObject io_object)
+{
+ bool no_temp_rel;
+
+ /*
+ * Some BackendTypes should never track IO statistics.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ return false;
+
+ /*
+ * Currently, IO on temporary relations can only occur in the
+ * IOCONTEXT_NORMAL IOContext.
+ */
+ if (io_context != IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * In core Postgres, only regular backends and WAL Sender processes
+ * executing queries will use local buffers and operate on temporary
+ * relations. Parallel workers will not use local buffers (see
+ * InitLocalBuffers()); however, extensions leveraging background workers
+ * have no such limitation, so track IO on IOOBJECT_TEMP_RELATION for
+ * BackendType B_BG_WORKER.
+ */
+ no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
+ bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
+
+ if (no_temp_rel && io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * Some BackendTypes do not currently perform any IO in certain
+ * IOContexts, and, while it may not be inherently incorrect for them to
+ * do so, excluding those rows from the view makes the view easier to use.
+ */
+ if ((bktype == B_CHECKPOINTER || bktype == B_BG_WRITER) &&
+ (io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE ||
+ io_context == IOCONTEXT_VACUUM))
+ return false;
+
+ if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
+ return false;
+
+ if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
+ io_context == IOCONTEXT_BULKWRITE)
+ return false;
+
+ return true;
+}
+
+/*
+ * Some BackendTypes will never do certain IOOps and some IOOps should not
+ * occur in certain IOContexts or on certain IOObjects. Check that the given
+ * IOOp is valid for the given BackendType in the given IOContext and on the
+ * given IOObject. Note that there are currently no cases of an IOOp being
+ * invalid for a particular BackendType only within a certain IOContext and/or
+ * only on a certain IOObject.
+ */
+bool
+pgstat_tracks_io_op(BackendType bktype, IOContext io_context,
+ IOObject io_object, IOOp io_op)
+{
+ bool strategy_io_context;
+
+ /* if (io_context, io_object) will never collect stats, we're done */
+ if (!pgstat_tracks_io_object(bktype, io_context, io_object))
+ return false;
+
+ /*
+ * Some BackendTypes will not do certain IOOps.
+ */
+ if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) &&
+ (io_op == IOOP_READ || io_op == IOOP_EVICT))
+ return false;
+
+ if ((bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER) && io_op == IOOP_EXTEND)
+ return false;
+
+ /*
+ * Some IOOps are not valid in certain IOContexts and some IOOps are only
+ * valid in certain contexts.
+ */
+ if (io_context == IOCONTEXT_BULKREAD && io_op == IOOP_EXTEND)
+ return false;
+
+ strategy_io_context = io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE || io_context == IOCONTEXT_VACUUM;
+
+ /*
+ * IOOP_REUSE is only relevant when a BufferAccessStrategy is in use.
+ */
+ if (!strategy_io_context && io_op == IOOP_REUSE)
+ return false;
+
+ /*
+ * IOOP_FSYNC IOOps done by a backend using a BufferAccessStrategy are
+ * counted in the IOCONTEXT_NORMAL IOContext. See comment in
+ * register_dirty_segment() for more details.
+ */
+ if (strategy_io_context && io_op == IOOP_FSYNC)
+ return false;
+
+ /*
+ * Temporary tables are not logged and thus do not require fsync'ing.
+ */
+ if (io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION && io_op == IOOP_FSYNC)
+ return false;
+
+ return true;
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 2e20b93c20..f793ac1516 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -206,7 +206,7 @@ pgstat_drop_relation(Relation rel)
}
/*
- * Report that the table was just vacuumed.
+ * Report that the table was just vacuumed and flush IO statistics.
*/
void
pgstat_report_vacuum(Oid tableoid, bool shared,
@@ -258,10 +258,18 @@ pgstat_report_vacuum(Oid tableoid, bool shared,
}
pgstat_unlock_entry(entry_ref);
+
+ /*
+ * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
+ * however this will not be called until after an entire autovacuum cycle
+ * is done -- which will likely vacuum many relations -- or until the
+ * VACUUM command has processed all tables and committed.
+ */
+ pgstat_flush_io(false);
}
/*
- * Report that the table was just analyzed.
+ * Report that the table was just analyzed and flush IO statistics.
*
* Caller must provide new live- and dead-tuples estimates, as well as a
* flag indicating whether to reset the mod_since_analyze counter.
@@ -341,6 +349,9 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
+
+ /* see pgstat_report_vacuum() */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index c1506b53d0..09fffd0e82 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -202,6 +202,10 @@ StatsShmemInit(void)
LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&ctl->io.locks[i],
+ LWTRANCHE_PGSTATS_DATA);
}
else
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index e7a82b5fed..e8598b2f4e 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -34,7 +34,7 @@ static WalUsage prevWalUsage;
/*
* Calculate how much WAL usage counters have increased and update
- * shared statistics.
+ * shared WAL and IO statistics.
*
* Must be called by processes that generate WAL, that do not call
* pgstat_report_stat(), like walwriter.
@@ -43,6 +43,8 @@ void
pgstat_report_wal(bool force)
{
pgstat_flush_wal(force);
+
+ pgstat_flush_io(force);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 58bd1360b9..6df9f06a20 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1576,7 +1576,12 @@ pg_stat_reset(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
-/* Reset some shared cluster-wide counters */
+/*
+ * Reset some shared cluster-wide counters
+ *
+ * When adding a new reset target, ideally the name should match that in
+ * pgstat_kind_infos, if relevant.
+ */
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
{
@@ -1593,6 +1598,8 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_BGWRITER);
pgstat_reset_of_kind(PGSTAT_KIND_CHECKPOINTER);
}
+ else if (strcmp(target, "io") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_IO);
else if (strcmp(target, "recovery_prefetch") == 0)
XLogPrefetchResetStats();
else if (strcmp(target, "wal") == 0)
@@ -1601,7 +1608,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"io\", \"recovery_prefetch\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 96b3a1e1a0..c309e0233d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -332,6 +332,8 @@ typedef enum BackendType
B_WAL_WRITER,
} BackendType;
+#define BACKEND_NUM_TYPES (B_WAL_WRITER + 1)
+
extern PGDLLIMPORT BackendType MyBackendType;
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5e3326a3b9..d0fa47ec07 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -48,6 +48,7 @@ typedef enum PgStat_Kind
PGSTAT_KIND_ARCHIVER,
PGSTAT_KIND_BGWRITER,
PGSTAT_KIND_CHECKPOINTER,
+ PGSTAT_KIND_IO,
PGSTAT_KIND_SLRU,
PGSTAT_KIND_WAL,
} PgStat_Kind;
@@ -276,6 +277,55 @@ typedef struct PgStat_CheckpointerStats
PgStat_Counter buf_fsync_backend;
} PgStat_CheckpointerStats;
+
+/*
+ * Types related to counting IO operations
+ */
+typedef enum IOContext
+{
+ IOCONTEXT_BULKREAD,
+ IOCONTEXT_BULKWRITE,
+ IOCONTEXT_NORMAL,
+ IOCONTEXT_VACUUM,
+} IOContext;
+
+#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
+#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
+
+typedef enum IOObject
+{
+ IOOBJECT_RELATION,
+ IOOBJECT_TEMP_RELATION,
+} IOObject;
+
+#define IOOBJECT_FIRST IOOBJECT_RELATION
+#define IOOBJECT_NUM_TYPES (IOOBJECT_TEMP_RELATION + 1)
+
+typedef enum IOOp
+{
+ IOOP_EVICT,
+ IOOP_EXTEND,
+ IOOP_FSYNC,
+ IOOP_READ,
+ IOOP_REUSE,
+ IOOP_WRITE,
+} IOOp;
+
+#define IOOP_FIRST IOOP_EVICT
+#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
+
+typedef struct PgStat_BktypeIO
+{
+ PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
+} PgStat_BktypeIO;
+
+typedef struct PgStat_IO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BktypeIO stats[BACKEND_NUM_TYPES];
+} PgStat_IO;
+
+
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -453,6 +503,24 @@ extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_bktype_io_stats_valid(PgStat_BktypeIO *context_ops,
+ BackendType bktype);
+extern void pgstat_count_io_op(IOOp io_op, IOObject io_object, IOContext io_context);
+extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern const char *pgstat_get_io_context_name(IOContext io_context);
+extern const char *pgstat_get_io_object_name(IOObject io_object);
+
+extern bool pgstat_tracks_io_bktype(BackendType bktype);
+extern bool pgstat_tracks_io_object(BackendType bktype,
+ IOContext io_context, IOObject io_object);
+extern bool pgstat_tracks_io_op(BackendType bktype, IOContext io_context,
+ IOObject io_object, IOOp io_op);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 12fd51f1ae..6badb2fde4 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -329,6 +329,17 @@ typedef struct PgStatShared_Checkpointer
PgStat_CheckpointerStats reset_offset;
} PgStatShared_Checkpointer;
+/* Shared-memory ready PgStat_IO */
+typedef struct PgStatShared_IO
+{
+ /*
+ * locks[i] protects stats.stats[i]. locks[0] also protects
+ * stats.stat_reset_timestamp.
+ */
+ LWLock locks[BACKEND_NUM_TYPES];
+ PgStat_IO stats;
+} PgStatShared_IO;
+
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -419,6 +430,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Archiver archiver;
PgStatShared_BgWriter bgwriter;
PgStatShared_Checkpointer checkpointer;
+ PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
} PgStat_ShmemControl;
@@ -442,6 +454,8 @@ typedef struct PgStat_Snapshot
PgStat_CheckpointerStats checkpointer;
+ PgStat_IO io;
+
PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS];
PgStat_WalStats wal;
@@ -549,6 +563,15 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_flush_io(bool nowait);
+extern void pgstat_io_reset_all_cb(TimestampTz ts);
+extern void pgstat_io_snapshot_cb(void);
+
+
/*
* Functions in pgstat_relation.c
*/
@@ -643,6 +666,13 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid)
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
+/*
+ * Variables in pgstat_io.c
+ */
+
+extern PGDLLIMPORT bool have_iostats;
+
+
/*
* Variables in pgstat_slru.c
*/
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 23bafec5f7..1be6e07980 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1106,7 +1106,10 @@ ID
INFIX
INT128
INTERFACE_INFO
+IOContext
IOFuncSelector
+IOObject
+IOOp
IPCompareMethod
ITEM
IV
@@ -2016,6 +2019,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2025,6 +2029,7 @@ PgStat_ArchiverStats
PgStat_BackendFunctionEntry
PgStat_BackendSubEntry
PgStat_BgWriterStats
+PgStat_BktypeIO
PgStat_CheckpointerStats
PgStat_Counter
PgStat_EntryRef
@@ -2033,6 +2038,7 @@ PgStat_FetchConsistency
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
+PgStat_IO
PgStat_Kind
PgStat_KindInfo
PgStat_LocalState
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-17 19:12 ` Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Andres Freund @ 2023-01-17 19:12 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; vignesh C <[email protected]>
Hi,
On 2023-01-17 12:22:14 -0500, Melanie Plageman wrote:
> > > @@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
> > > .snapshot_cb = pgstat_checkpointer_snapshot_cb,
> > > },
> > >
> > > + [PGSTAT_KIND_IO] = {
> > > + .name = "io_ops",
> >
> > That should be "io" now I think?
> >
>
> Oh no! I didn't notice this was broken. I've added pg_stat_have_stats()
> to the IO stats tests now.
>
> It would be nice if pgstat_get_kind_from_str() could be used in
> pg_stat_reset_shared() to avoid having to remember to change both.
It's hard to make that work, because of the historical behaviour of that
function :(
> Also:
> - Since recovery_prefetch doesn't have a statistic kind, it doesn't fit
> well into this paradigm
I think that needs a rework anyway - it went in at about the same time as the
shared mem stats patch, so it doesn't quite cohere.
> On a separate note, should we be setting have_[io/slru/etc]stats to
> false in the reset all functions?
That'd not work reliably, because other backends won't do the same. I don't
see a benefit in doing it differently in the local connection than the other
connections.
> > > +typedef struct PgStat_BackendIO
> > > +{
> > > + PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
> > > +} PgStat_BackendIO;
> >
> > Would it bother you if we swapped the order of iocontext and iobject here and
> > related places? It makes more sense to me semantically, and should now be
> > pretty easy, code wise.
>
> So, thinking about this I started noticing inconsistencies in other
> areas around this order:
> For example: ordering of objects mentioned in commit messages and comments,
> ordering of parameters (like in pgstat_count_io_op() [currently in
> reverse order]).
>
> I think we should make a final decision about this ordering and then
> make everywhere consistent (including ordering in the view).
>
> Currently the order is:
> BackendType
> IOContext
> IOObject
> IOOp
>
> You are suggesting this order:
> BackendType
> IOObject
> IOContext
> IOOp
>
> Could you explain what you find more natural about this ordering (as I
> find the other more natural)?
The object we're performing IO on determines more things than the context. So
it just seems like the natural hierarchical fit. The context is a sub-category
of the object. Consider how it'll look like if we also have objects for 'wal',
'temp files'. It'll make sense to group by just the object, but it won't make
sense to group by just the context.
If it were trivial to do I'd use a different IOContext for each IOObject. But
it'd make it much harder. So there'll just be a bunch of values of IOContext
that'll only be used for one or a subset of the IOObjects.
The reason to put BackendType at the top is pragmatic - one backend is of a
single type, but can do IO for all kinds of objects/contexts. So any other
hierarchy would make the locking etc much harder.
> This is one possible natural sentence with these objects:
>
> During COPY, a client backend may read in data from a permanent
> relation.
> This order is:
> IOContext
> BackendType
> IOOp
> IOObject
>
> I think English sentences are often structured subject, verb, object --
> but in our case, we have an extra thing that doesn't fit neatly
> (IOContext).
"..., to avoid polluting the buffer cache it uses the bulk (read|write)
strategy".
> Also, IOOp in a sentence would be in the middle (as the
> verb). I made it last because a) it feels like the smallest unit b) it
> would make the code a lot more annoying if it wasn't last.
Yea, I think pragmatically that is the right choice.
> > > Subject: [PATCH v47 3/5] pgstat: Count IO for relations
> >
> > Nearly happy with this now. See one minor nit below.
> >
> > I don't love the counting in register_dirty_segment() and mdsyncfiletag(), but
> > I don't have a better idea, and it doesn't seem too horrible.
>
> You don't like it because such things shouldn't be in md.c -- since we
> went to the trouble of having function pointers and making it general?
It's more of a gut feeling than well reasoned ;)
> > > +-- Change the tablespace so that the table is rewritten directly, then SELECT
> > > +-- from it to cause it to be read back into shared buffers.
> > > +SET allow_in_place_tablespaces = true;
> > > +CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
> >
> > Perhaps worth doing this in tablespace.sql, to avoid the additional
> > checkpoints done as part of CREATE/DROP TABLESPACE?
> >
> > Or, at least combine this with the CHECKPOINTs above?
>
> I see a checkpoint is requested when dropping the tablespace if not all
> the files in it are deleted. It seems like if the DROP TABLE for the
> permanent table is before the explicit checkpoints in the test, then the
> DROP TABLESPACE will not cause an additional checkpoint.
Unfortunately, that's not how it works :(. See the comment above mdunlink():
> * For regular relations, we don't unlink the first segment file of the rel,
> * but just truncate it to zero length, and record a request to unlink it after
> * the next checkpoint. Additional segments can be unlinked immediately,
> * however. Leaving the empty file in place prevents that relfilenumber
> * from being reused. The scenario this protects us from is:
> ...
> Is this what you are suggesting? Dropping the temporary table should not
> have an effect on this.
I was wondering about simply moving that portion of the test to
tablespace.sql, where we already created a tablespace.
An alternative would be to propose splitting tablespace.sql into one portion
running at the start of parallel_schedule, and one at the end. Historically,
we needed tablespace.sql to be optional due to causing problems when
replicating to another instance on the same machine, but now we have
allow_in_place_tablespaces.
> > SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
> >
> > Better toast compression or such could easily make test_io_local smaller than
> > it's today. Seeing that it's too small would make it easier to understand the
> > failure.
>
> Good idea. So, I used pg_table_size() because it seems like
> pg_relation_size() does not include the toast relations. However, I'm
> not sure this is a good idea, because pg_table_size() includes FSM and
> visibility map. Should I write a query to get the toast relation name
> and add pg_relation_size() of that relation and the main relation?
I think it's the right thing to just include the relation size. Your queries
IIRC won't use the toast table or other forks. So I'd leave it at just
pg_relation_size().
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-01-17 22:00 ` Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Melanie Plageman @ 2023-01-17 22:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; vignesh C <[email protected]>
v49 attached
On Tue, Jan 17, 2023 at 2:12 PM Andres Freund <[email protected]> wrote:
> On 2023-01-17 12:22:14 -0500, Melanie Plageman wrote:
>
> > > > +typedef struct PgStat_BackendIO
> > > > +{
> > > > + PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
> > > > +} PgStat_BackendIO;
> > >
> > > Would it bother you if we swapped the order of iocontext and iobject here and
> > > related places? It makes more sense to me semantically, and should now be
> > > pretty easy, code wise.
> >
> > So, thinking about this I started noticing inconsistencies in other
> > areas around this order:
> > For example: ordering of objects mentioned in commit messages and comments,
> > ordering of parameters (like in pgstat_count_io_op() [currently in
> > reverse order]).
> >
> > I think we should make a final decision about this ordering and then
> > make everywhere consistent (including ordering in the view).
> >
> > Currently the order is:
> > BackendType
> > IOContext
> > IOObject
> > IOOp
> >
> > You are suggesting this order:
> > BackendType
> > IOObject
> > IOContext
> > IOOp
> >
> > Could you explain what you find more natural about this ordering (as I
> > find the other more natural)?
>
> The object we're performing IO on determines more things than the context. So
> it just seems like the natural hierarchical fit. The context is a sub-category
> of the object. Consider how it'll look like if we also have objects for 'wal',
> 'temp files'. It'll make sense to group by just the object, but it won't make
> sense to group by just the context.
>
> If it were trivial to do I'd use a different IOContext for each IOObject. But
> it'd make it much harder. So there'll just be a bunch of values of IOContext
> that'll only be used for one or a subset of the IOObjects.
>
>
> The reason to put BackendType at the top is pragmatic - one backend is of a
> single type, but can do IO for all kinds of objects/contexts. So any other
> hierarchy would make the locking etc much harder.
>
>
> > This is one possible natural sentence with these objects:
> >
> > During COPY, a client backend may read in data from a permanent
> > relation.
> > This order is:
> > IOContext
> > BackendType
> > IOOp
> > IOObject
> >
> > I think English sentences are often structured subject, verb, object --
> > but in our case, we have an extra thing that doesn't fit neatly
> > (IOContext).
>
> "..., to avoid polluting the buffer cache it uses the bulk (read|write)
> strategy".
>
>
> > Also, IOOp in a sentence would be in the middle (as the
> > verb). I made it last because a) it feels like the smallest unit b) it
> > would make the code a lot more annoying if it wasn't last.
>
> Yea, I think pragmatically that is the right choice.
I have changed the order and updated all the places using
PgStat_BktypeIO as well as in all locations in which it should be
ordered for consistency (that I could find in the pass I did) -- e.g.
the view definition, function signatures, comments, commit messages,
etc.
> > > > +-- Change the tablespace so that the table is rewritten directly, then SELECT
> > > > +-- from it to cause it to be read back into shared buffers.
> > > > +SET allow_in_place_tablespaces = true;
> > > > +CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
> > >
> > > Perhaps worth doing this in tablespace.sql, to avoid the additional
> > > checkpoints done as part of CREATE/DROP TABLESPACE?
> > >
> > > Or, at least combine this with the CHECKPOINTs above?
> >
> > I see a checkpoint is requested when dropping the tablespace if not all
> > the files in it are deleted. It seems like if the DROP TABLE for the
> > permanent table is before the explicit checkpoints in the test, then the
> > DROP TABLESPACE will not cause an additional checkpoint.
>
> Unfortunately, that's not how it works :(. See the comment above mdunlink():
>
> > * For regular relations, we don't unlink the first segment file of the rel,
> > * but just truncate it to zero length, and record a request to unlink it after
> > * the next checkpoint. Additional segments can be unlinked immediately,
> > * however. Leaving the empty file in place prevents that relfilenumber
> > * from being reused. The scenario this protects us from is:
> > ...
>
>
> > Is this what you are suggesting? Dropping the temporary table should not
> > have an effect on this.
>
> I was wondering about simply moving that portion of the test to
> tablespace.sql, where we already created a tablespace.
>
>
> An alternative would be to propose splitting tablespace.sql into one portion
> running at the start of parallel_schedule, and one at the end. Historically,
> we needed tablespace.sql to be optional due to causing problems when
> replicating to another instance on the same machine, but now we have
> allow_in_place_tablespaces.
It seems like the best way would be to split up the tablespace test file
as you suggested and drop the tablespace at the end of the regression
test suite. There could be other tests that could use a tablespace.
Though what I wrote is kind of tablespace test coverage, if this
rewriting behavior no longer happened when doing alter table set
tablespace, we would want to come up with a new test which exercised
that code to count those IO stats, not simply delete it from the
tablespace tests.
> > > SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
> > >
> > > Better toast compression or such could easily make test_io_local smaller than
> > > it's today. Seeing that it's too small would make it easier to understand the
> > > failure.
> >
> > Good idea. So, I used pg_table_size() because it seems like
> > pg_relation_size() does not include the toast relations. However, I'm
> > not sure this is a good idea, because pg_table_size() includes FSM and
> > visibility map. Should I write a query to get the toast relation name
> > and add pg_relation_size() of that relation and the main relation?
>
> I think it's the right thing to just include the relation size. Your queries
> IIRC won't use the toast table or other forks. So I'd leave it at just
> pg_relation_size().
I did notice that this test wasn't using the toast table for the
toastable column -- but you mentioned better toast compression affecting
the future test stability, so I'm confused.
- Melanie
Attachments:
[text/x-patch] v49-0001-pgstat-Infrastructure-to-track-IO-operations.patch (28.6K, ../../CAAKRu_YJpZ_fV_n5PdOLWp9NcDZP2n=OZyLSUQRwBCz0iYpK_g@mail.gmail.com/2-v49-0001-pgstat-Infrastructure-to-track-IO-operations.patch)
download | inline diff:
From 2e29ec2d41fee3fd299c271ade82f8270a16474b Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 17 Jan 2023 16:10:34 -0500
Subject: [PATCH v49 1/4] pgstat: Infrastructure to track IO operations
Introduce "IOOp", an IO operation done by a backend, "IOObject", the
target object of the IO, and "IOContext", the context or location of the
IO operations on that object. For example, the checkpointer may write a
shared buffer out. This would be considered an IOOP_WRITE IOOp on an
IOOBJECT_RELATION IOObject in the IOCONTEXT_NORMAL IOContext by
BackendType B_CHECKPOINTER.
Each BackendType counts IOOps (evict, extend, fsync, read, reuse, and
write) per IOObject (relation, temp relation) per IOContext (normal,
bulkread, bulkwrite, or vacuum) through a call to pgstat_count_io_op().
Note that this commit introduces the infrastructure to count IO
Operation statistics. A subsequent commit will add calls to
pgstat_count_io_op() in the appropriate locations.
IOObject IOOBJECT_TEMP_RELATION concerns IO Operations on buffers
containing temporary table data, while IOObject IOOBJECT_RELATION
concerns IO Operations on buffers containing permanent relation data.
IOContext IOCONTEXT_NORMAL concerns operations on local and shared
buffers, while IOCONTEXT_BULKREAD, IOCONTEXT_BULKWRITE, and
IOCONTEXT_VACUUM IOContexts concern IO operations on buffers as part of
a BufferAccessStrategy.
Stats on IOOps on all IOObjects in all IOContexts for a given backend
are first counted in a backend's local memory and then flushed to shared
memory and accumulated with those from all other backends, exited and
live.
Some BackendTypes will not flush their pending statistics at regular
intervals and explicitly call pgstat_flush_io_ops() during the course of
normal operations to flush their backend-local IO operation statistics
to shared memory in a timely manner.
Because not all BackendType, IOObject, IOContext, IOOp combinations are
valid, the validity of the stats is checked before flushing pending
stats and before reading in the existing stats file to shared memory.
The aggregated stats in shared memory could be extended in the future
with per-backend stats -- useful for per connection IO statistics and
monitoring.
PGSTAT_FILE_FORMAT_ID should be bumped with this commit.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Reviewed-by: Kyotaro Horiguchi <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 2 +
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 26 ++
src/backend/utils/activity/pgstat_bgwriter.c | 7 +-
.../utils/activity/pgstat_checkpointer.c | 7 +-
src/backend/utils/activity/pgstat_io.c | 386 ++++++++++++++++++
src/backend/utils/activity/pgstat_relation.c | 15 +-
src/backend/utils/activity/pgstat_shmem.c | 4 +
src/backend/utils/activity/pgstat_wal.c | 4 +-
src/backend/utils/adt/pgstatfuncs.c | 11 +-
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 68 +++
src/include/utils/pgstat_internal.h | 30 ++
src/tools/pgindent/typedefs.list | 6 +
15 files changed, 563 insertions(+), 7 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_io.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 358d2ff90f..8d51ca3773 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5418,6 +5418,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
the <structname>pg_stat_bgwriter</structname>
view, <literal>archiver</literal> to reset all the counters shown in
the <structname>pg_stat_archiver</structname> view,
+ <literal>io</literal> to reset all the counters shown in the
+ <structname>pg_stat_io</structname> view,
<literal>wal</literal> to reset all the counters shown in the
<structname>pg_stat_wal</structname> view or
<literal>recovery_prefetch</literal> to reset all the counters shown
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index a80eda3cf4..7d7482dde0 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -22,6 +22,7 @@ OBJS = \
pgstat_checkpointer.o \
pgstat_database.o \
pgstat_function.o \
+ pgstat_io.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index a2b872c24b..518ee3f798 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -9,6 +9,7 @@ backend_sources += files(
'pgstat_checkpointer.c',
'pgstat_database.c',
'pgstat_function.c',
+ 'pgstat_io.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 0fa5370bcd..60fc4e761f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -72,6 +72,7 @@
* - pgstat_checkpointer.c
* - pgstat_database.c
* - pgstat_function.c
+ * - pgstat_io.c
* - pgstat_relation.c
* - pgstat_replslot.c
* - pgstat_slru.c
@@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
+ [PGSTAT_KIND_IO] = {
+ .name = "io",
+
+ .fixed_amount = true,
+
+ .reset_all_cb = pgstat_io_reset_all_cb,
+ .snapshot_cb = pgstat_io_snapshot_cb,
+ },
+
[PGSTAT_KIND_SLRU] = {
.name = "slru",
@@ -582,6 +592,7 @@ pgstat_report_stat(bool force)
/* Don't expend a clock check if nothing to do */
if (dlist_is_empty(&pgStatPending) &&
+ !have_iostats &&
!have_slrustats &&
!pgstat_have_pending_wal())
{
@@ -628,6 +639,9 @@ pgstat_report_stat(bool force)
/* flush database / relation / function / ... stats */
partial_flush |= pgstat_flush_pending_entries(nowait);
+ /* flush IO stats */
+ partial_flush |= pgstat_flush_io(nowait);
+
/* flush wal stats */
partial_flush |= pgstat_flush_wal(nowait);
@@ -1322,6 +1336,12 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
+ /*
+ * Write IO stats struct
+ */
+ pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
+ write_chunk_s(fpout, &pgStatLocal.snapshot.io);
+
/*
* Write SLRU stats struct
*/
@@ -1496,6 +1516,12 @@ pgstat_read_statsfile(void)
if (!read_chunk_s(fpin, &shmem->checkpointer.stats))
goto error;
+ /*
+ * Read IO stats struct
+ */
+ if (!read_chunk_s(fpin, &shmem->io.stats))
+ goto error;
+
/*
* Read SLRU stats struct
*/
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 9247f2dda2..92be384b0d 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -24,7 +24,7 @@ PgStat_BgWriterStats PendingBgWriterStats = {0};
/*
- * Report bgwriter statistics
+ * Report bgwriter and IO statistics
*/
void
pgstat_report_bgwriter(void)
@@ -56,6 +56,11 @@ pgstat_report_bgwriter(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 3e9ab45103..26dec112f6 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -24,7 +24,7 @@ PgStat_CheckpointerStats PendingCheckpointerStats = {0};
/*
- * Report checkpointer statistics
+ * Report checkpointer and IO statistics
*/
void
pgstat_report_checkpointer(void)
@@ -62,6 +62,11 @@ pgstat_report_checkpointer(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingCheckpointerStats, 0, sizeof(PendingCheckpointerStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
new file mode 100644
index 0000000000..b606f23eb8
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -0,0 +1,386 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_io.c
+ * Implementation of IO statistics.
+ *
+ * This file contains the implementation of IO statistics. It is kept separate
+ * from pgstat.c to enforce the line between the statistics access / storage
+ * implementation and the details about individual types of statistics.
+ *
+ * Copyright (c) 2021-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_io.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "utils/pgstat_internal.h"
+
+
+static PgStat_BktypeIO PendingIOStats;
+bool have_iostats = false;
+
+/*
+ * Check that stats have not been counted for any combination of IOObject,
+ * IOContext, and IOOp which are not tracked for the passed-in BackendType. The
+ * passed-in PgStat_BktypeIO must contain stats from the BackendType specified
+ * by the second parameter. Caller is responsible for locking the passed-in
+ * PgStat_BktypeIO, if needed.
+ */
+bool
+pgstat_bktype_io_stats_valid(PgStat_BktypeIO *backend_io,
+ BackendType bktype)
+{
+ bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ /*
+ * Don't bother trying to skip to the next loop iteration if
+ * pgstat_tracks_io_object() would return false here. We still
+ * need to validate that each counter is zero anyway.
+ */
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ /* No stats, so nothing to validate */
+ if (backend_io->data[io_object][io_context][io_op] == 0)
+ continue;
+
+ /* There are stats and there shouldn't be */
+ if (!bktype_tracked ||
+ !pgstat_tracks_io_op(bktype, io_object, io_context, io_op))
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
+void
+pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op)
+{
+ Assert(io_object < IOOBJECT_NUM_TYPES);
+ Assert(io_context < IOCONTEXT_NUM_TYPES);
+ Assert(io_op < IOOP_NUM_TYPES);
+ Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
+
+ PendingIOStats.data[io_object][io_context][io_op]++;
+
+ have_iostats = true;
+}
+
+PgStat_IO *
+pgstat_fetch_stat_io(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_IO);
+
+ return &pgStatLocal.snapshot.io;
+}
+
+/*
+ * Flush out locally pending IO statistics
+ *
+ * If no stats have been recorded, this function returns false.
+ *
+ * If nowait is true, this function returns true if the lock could not be
+ * acquired. Otherwise, return false.
+ */
+bool
+pgstat_flush_io(bool nowait)
+{
+ LWLock *bktype_lock;
+ PgStat_BktypeIO *bktype_shstats;
+
+ if (!have_iostats)
+ return false;
+
+ bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
+ bktype_shstats =
+ &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+
+ if (!nowait)
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
+ return true;
+
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOOp io_op = IOOP_FIRST;
+ io_op < IOOP_NUM_TYPES; io_op++)
+ bktype_shstats->data[io_object][io_context][io_op] +=
+ PendingIOStats.data[io_object][io_context][io_op];
+
+ Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
+
+ LWLockRelease(bktype_lock);
+
+ memset(&PendingIOStats, 0, sizeof(PendingIOStats));
+
+ have_iostats = false;
+
+ return false;
+}
+
+const char *
+pgstat_get_io_context_name(IOContext io_context)
+{
+ switch (io_context)
+ {
+ case IOCONTEXT_BULKREAD:
+ return "bulkread";
+ case IOCONTEXT_BULKWRITE:
+ return "bulkwrite";
+ case IOCONTEXT_NORMAL:
+ return "normal";
+ case IOCONTEXT_VACUUM:
+ return "vacuum";
+ }
+
+ elog(ERROR, "unrecognized IOContext value: %d", io_context);
+ pg_unreachable();
+}
+
+const char *
+pgstat_get_io_object_name(IOObject io_object)
+{
+ switch (io_object)
+ {
+ case IOOBJECT_RELATION:
+ return "relation";
+ case IOOBJECT_TEMP_RELATION:
+ return "temp relation";
+ }
+
+ elog(ERROR, "unrecognized IOObject value: %d", io_object);
+ pg_unreachable();
+}
+
+void
+pgstat_io_reset_all_cb(TimestampTz ts)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+
+ memset(bktype_shstats, 0, sizeof(*bktype_shstats));
+ LWLockRelease(bktype_lock);
+ }
+}
+
+void
+pgstat_io_snapshot_cb(void)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_SHARED);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.snapshot.io.stat_reset_timestamp =
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+
+ /* using struct assignment due to better type safety */
+ *bktype_snap = *bktype_shstats;
+ LWLockRelease(bktype_lock);
+ }
+}
+
+/*
+* IO statistics are not collected for all BackendTypes.
+*
+* The following BackendTypes do not participate in the cumulative stats
+* subsystem or do not perform IO on which we currently track:
+* - Syslogger because it is not connected to shared memory
+* - Archiver because most relevant archiving IO is delegated to a
+* specialized command or module
+* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
+*
+* Function returns true if BackendType participates in the cumulative stats
+* subsystem for IO and false if it does not.
+*
+* When adding a new BackendType, also consider adding relevant restrictions to
+* pgstat_tracks_io_object() and pgstat_tracks_io_op().
+*/
+bool
+pgstat_tracks_io_bktype(BackendType bktype)
+{
+ /*
+ * List every type so that new backend types trigger a warning about
+ * needing to adjust this switch.
+ */
+ switch (bktype)
+ {
+ case B_INVALID:
+ case B_ARCHIVER:
+ case B_LOGGER:
+ case B_WAL_RECEIVER:
+ case B_WAL_WRITER:
+ return false;
+
+ case B_AUTOVAC_LAUNCHER:
+ case B_AUTOVAC_WORKER:
+ case B_BACKEND:
+ case B_BG_WORKER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_STANDALONE_BACKEND:
+ case B_STARTUP:
+ case B_WAL_SENDER:
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Some BackendTypes do not perform IO on certain IOObjects or in certain
+ * IOContexts. Some IOObjects are never operated on in some IOContexts. Check
+ * that the given BackendType is expected to do IO in the given IOContext and
+ * on the given IOObject and that the given IOObject is expected to be operated
+ * on in the given IOContext.
+ */
+bool
+pgstat_tracks_io_object(BackendType bktype, IOObject io_object,
+ IOContext io_context)
+{
+ bool no_temp_rel;
+
+ /*
+ * Some BackendTypes should never track IO statistics.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ return false;
+
+ /*
+ * Currently, IO on temporary relations can only occur in the
+ * IOCONTEXT_NORMAL IOContext.
+ */
+ if (io_context != IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * In core Postgres, only regular backends and WAL Sender processes
+ * executing queries will use local buffers and operate on temporary
+ * relations. Parallel workers will not use local buffers (see
+ * InitLocalBuffers()); however, extensions leveraging background workers
+ * have no such limitation, so track IO on IOOBJECT_TEMP_RELATION for
+ * BackendType B_BG_WORKER.
+ */
+ no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
+ bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
+
+ if (no_temp_rel && io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * Some BackendTypes do not currently perform any IO in certain
+ * IOContexts, and, while it may not be inherently incorrect for them to
+ * do so, excluding those rows from the view makes the view easier to use.
+ */
+ if ((bktype == B_CHECKPOINTER || bktype == B_BG_WRITER) &&
+ (io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE ||
+ io_context == IOCONTEXT_VACUUM))
+ return false;
+
+ if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
+ return false;
+
+ if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
+ io_context == IOCONTEXT_BULKWRITE)
+ return false;
+
+ return true;
+}
+
+/*
+ * Some BackendTypes will never do certain IOOps and some IOOps should not
+ * occur in certain IOContexts or on certain IOObjects. Check that the given
+ * IOOp is valid for the given BackendType in the given IOContext and on the
+ * given IOObject. Note that there are currently no cases of an IOOp being
+ * invalid for a particular BackendType only within a certain IOContext and/or
+ * only on a certain IOObject.
+ */
+bool
+pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
+ IOContext io_context, IOOp io_op)
+{
+ bool strategy_io_context;
+
+ /* if (io_context, io_object) will never collect stats, we're done */
+ if (!pgstat_tracks_io_object(bktype, io_object, io_context))
+ return false;
+
+ /*
+ * Some BackendTypes will not do certain IOOps.
+ */
+ if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) &&
+ (io_op == IOOP_READ || io_op == IOOP_EVICT))
+ return false;
+
+ if ((bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER) && io_op == IOOP_EXTEND)
+ return false;
+
+ /*
+ * Some IOOps are not valid in certain IOContexts and some IOOps are only
+ * valid in certain contexts.
+ */
+ if (io_context == IOCONTEXT_BULKREAD && io_op == IOOP_EXTEND)
+ return false;
+
+ strategy_io_context = io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE || io_context == IOCONTEXT_VACUUM;
+
+ /*
+ * IOOP_REUSE is only relevant when a BufferAccessStrategy is in use.
+ */
+ if (!strategy_io_context && io_op == IOOP_REUSE)
+ return false;
+
+ /*
+ * IOOP_FSYNC IOOps done by a backend using a BufferAccessStrategy are
+ * counted in the IOCONTEXT_NORMAL IOContext. See comment in
+ * register_dirty_segment() for more details.
+ */
+ if (strategy_io_context && io_op == IOOP_FSYNC)
+ return false;
+
+ /*
+ * Temporary tables are not logged and thus do not require fsync'ing.
+ */
+ if (io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION && io_op == IOOP_FSYNC)
+ return false;
+
+ return true;
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 2e20b93c20..f793ac1516 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -206,7 +206,7 @@ pgstat_drop_relation(Relation rel)
}
/*
- * Report that the table was just vacuumed.
+ * Report that the table was just vacuumed and flush IO statistics.
*/
void
pgstat_report_vacuum(Oid tableoid, bool shared,
@@ -258,10 +258,18 @@ pgstat_report_vacuum(Oid tableoid, bool shared,
}
pgstat_unlock_entry(entry_ref);
+
+ /*
+ * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
+ * however this will not be called until after an entire autovacuum cycle
+ * is done -- which will likely vacuum many relations -- or until the
+ * VACUUM command has processed all tables and committed.
+ */
+ pgstat_flush_io(false);
}
/*
- * Report that the table was just analyzed.
+ * Report that the table was just analyzed and flush IO statistics.
*
* Caller must provide new live- and dead-tuples estimates, as well as a
* flag indicating whether to reset the mod_since_analyze counter.
@@ -341,6 +349,9 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
+
+ /* see pgstat_report_vacuum() */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index c1506b53d0..09fffd0e82 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -202,6 +202,10 @@ StatsShmemInit(void)
LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&ctl->io.locks[i],
+ LWTRANCHE_PGSTATS_DATA);
}
else
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index e7a82b5fed..e8598b2f4e 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -34,7 +34,7 @@ static WalUsage prevWalUsage;
/*
* Calculate how much WAL usage counters have increased and update
- * shared statistics.
+ * shared WAL and IO statistics.
*
* Must be called by processes that generate WAL, that do not call
* pgstat_report_stat(), like walwriter.
@@ -43,6 +43,8 @@ void
pgstat_report_wal(bool force)
{
pgstat_flush_wal(force);
+
+ pgstat_flush_io(force);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 58bd1360b9..6df9f06a20 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1576,7 +1576,12 @@ pg_stat_reset(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
-/* Reset some shared cluster-wide counters */
+/*
+ * Reset some shared cluster-wide counters
+ *
+ * When adding a new reset target, ideally the name should match that in
+ * pgstat_kind_infos, if relevant.
+ */
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
{
@@ -1593,6 +1598,8 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_BGWRITER);
pgstat_reset_of_kind(PGSTAT_KIND_CHECKPOINTER);
}
+ else if (strcmp(target, "io") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_IO);
else if (strcmp(target, "recovery_prefetch") == 0)
XLogPrefetchResetStats();
else if (strcmp(target, "wal") == 0)
@@ -1601,7 +1608,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"io\", \"recovery_prefetch\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 96b3a1e1a0..c309e0233d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -332,6 +332,8 @@ typedef enum BackendType
B_WAL_WRITER,
} BackendType;
+#define BACKEND_NUM_TYPES (B_WAL_WRITER + 1)
+
extern PGDLLIMPORT BackendType MyBackendType;
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5e3326a3b9..9f09caa05f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -48,6 +48,7 @@ typedef enum PgStat_Kind
PGSTAT_KIND_ARCHIVER,
PGSTAT_KIND_BGWRITER,
PGSTAT_KIND_CHECKPOINTER,
+ PGSTAT_KIND_IO,
PGSTAT_KIND_SLRU,
PGSTAT_KIND_WAL,
} PgStat_Kind;
@@ -276,6 +277,55 @@ typedef struct PgStat_CheckpointerStats
PgStat_Counter buf_fsync_backend;
} PgStat_CheckpointerStats;
+
+/*
+ * Types related to counting IO operations
+ */
+typedef enum IOObject
+{
+ IOOBJECT_RELATION,
+ IOOBJECT_TEMP_RELATION,
+} IOObject;
+
+#define IOOBJECT_FIRST IOOBJECT_RELATION
+#define IOOBJECT_NUM_TYPES (IOOBJECT_TEMP_RELATION + 1)
+
+typedef enum IOContext
+{
+ IOCONTEXT_BULKREAD,
+ IOCONTEXT_BULKWRITE,
+ IOCONTEXT_NORMAL,
+ IOCONTEXT_VACUUM,
+} IOContext;
+
+#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
+#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
+
+typedef enum IOOp
+{
+ IOOP_EVICT,
+ IOOP_EXTEND,
+ IOOP_FSYNC,
+ IOOP_READ,
+ IOOP_REUSE,
+ IOOP_WRITE,
+} IOOp;
+
+#define IOOP_FIRST IOOP_EVICT
+#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
+
+typedef struct PgStat_BktypeIO
+{
+ PgStat_Counter data[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
+} PgStat_BktypeIO;
+
+typedef struct PgStat_IO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BktypeIO stats[BACKEND_NUM_TYPES];
+} PgStat_IO;
+
+
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -453,6 +503,24 @@ extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_bktype_io_stats_valid(PgStat_BktypeIO *context_ops,
+ BackendType bktype);
+extern void pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op);
+extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern const char *pgstat_get_io_context_name(IOContext io_context);
+extern const char *pgstat_get_io_object_name(IOObject io_object);
+
+extern bool pgstat_tracks_io_bktype(BackendType bktype);
+extern bool pgstat_tracks_io_object(BackendType bktype,
+ IOObject io_object, IOContext io_context);
+extern bool pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
+ IOContext io_context, IOOp io_op);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 12fd51f1ae..6badb2fde4 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -329,6 +329,17 @@ typedef struct PgStatShared_Checkpointer
PgStat_CheckpointerStats reset_offset;
} PgStatShared_Checkpointer;
+/* Shared-memory ready PgStat_IO */
+typedef struct PgStatShared_IO
+{
+ /*
+ * locks[i] protects stats.stats[i]. locks[0] also protects
+ * stats.stat_reset_timestamp.
+ */
+ LWLock locks[BACKEND_NUM_TYPES];
+ PgStat_IO stats;
+} PgStatShared_IO;
+
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -419,6 +430,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Archiver archiver;
PgStatShared_BgWriter bgwriter;
PgStatShared_Checkpointer checkpointer;
+ PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
} PgStat_ShmemControl;
@@ -442,6 +454,8 @@ typedef struct PgStat_Snapshot
PgStat_CheckpointerStats checkpointer;
+ PgStat_IO io;
+
PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS];
PgStat_WalStats wal;
@@ -549,6 +563,15 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_flush_io(bool nowait);
+extern void pgstat_io_reset_all_cb(TimestampTz ts);
+extern void pgstat_io_snapshot_cb(void);
+
+
/*
* Functions in pgstat_relation.c
*/
@@ -643,6 +666,13 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid)
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
+/*
+ * Variables in pgstat_io.c
+ */
+
+extern PGDLLIMPORT bool have_iostats;
+
+
/*
* Variables in pgstat_slru.c
*/
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 23bafec5f7..1be6e07980 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1106,7 +1106,10 @@ ID
INFIX
INT128
INTERFACE_INFO
+IOContext
IOFuncSelector
+IOObject
+IOOp
IPCompareMethod
ITEM
IV
@@ -2016,6 +2019,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2025,6 +2029,7 @@ PgStat_ArchiverStats
PgStat_BackendFunctionEntry
PgStat_BackendSubEntry
PgStat_BgWriterStats
+PgStat_BktypeIO
PgStat_CheckpointerStats
PgStat_Counter
PgStat_EntryRef
@@ -2033,6 +2038,7 @@ PgStat_FetchConsistency
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
+PgStat_IO
PgStat_Kind
PgStat_KindInfo
PgStat_LocalState
--
2.34.1
[text/x-patch] v49-0004-pg_stat_io-documentation.patch (13.6K, ../../CAAKRu_YJpZ_fV_n5PdOLWp9NcDZP2n=OZyLSUQRwBCz0iYpK_g@mail.gmail.com/3-v49-0004-pg_stat_io-documentation.patch)
download | inline diff:
From 86be2a8ef4e800061ca57f0ba42ac4ebc0c4ac91 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 17 Jan 2023 16:34:27 -0500
Subject: [PATCH v49 4/4] pg_stat_io documentation
Author: Melanie Plageman <[email protected]>
Author: Samay Sharma <[email protected]>
Reviewed-by: Maciek Sakrejda <[email protected]>
Reviewed-by: Lukas Fittl <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 321 +++++++++++++++++++++++++++++++++--
1 file changed, 307 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 8d51ca3773..b875fc3f12 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -469,6 +469,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
+ <entry>
+ One row for each combination of backend type, context, and target object
+ containing cluster-wide I/O statistics.
+ See <link linkend="monitoring-pg-stat-io-view">
+ <structname>pg_stat_io</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_replication_slots</structname><indexterm><primary>pg_stat_replication_slots</primary></indexterm></entry>
<entry>One row per replication slot, showing statistics about the
@@ -665,20 +675,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</para>
<para>
- The <structname>pg_statio_</structname> views are primarily useful to
- determine the effectiveness of the buffer cache. When the number
- of actual disk reads is much smaller than the number of buffer
- hits, then the cache is satisfying most read requests without
- invoking a kernel call. However, these statistics do not give the
- entire story: due to the way in which <productname>PostgreSQL</productname>
- handles disk I/O, data that is not in the
- <productname>PostgreSQL</productname> buffer cache might still reside in the
- kernel's I/O cache, and might therefore still be fetched without
- requiring a physical read. Users interested in obtaining more
- detailed information on <productname>PostgreSQL</productname> I/O behavior are
- advised to use the <productname>PostgreSQL</productname> statistics views
- in combination with operating system utilities that allow insight
- into the kernel's handling of I/O.
+ The <structname>pg_stat_io</structname> and
+ <structname>pg_statio_</structname> set of views are useful for determining
+ the effectiveness of the buffer cache. They can be used to calculate a cache
+ hit ratio. Note that while <productname>PostgreSQL</productname>'s I/O
+ statistics capture most instances in which the kernel was invoked in order
+ to perform I/O, they do not differentiate between data which had to be
+ fetched from disk and that which already resided in the kernel page cache.
+ Users are advised to use the <productname>PostgreSQL</productname>
+ statistics views in combination with operating system utilities for a more
+ complete picture of their database's I/O performance.
</para>
</sect2>
@@ -3643,6 +3649,293 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_archived_wal</structfield> have also been successfully
archived.
</para>
+ </sect2>
+
+ <sect2 id="monitoring-pg-stat-io-view">
+ <title><structname>pg_stat_io</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_io</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_io</structname> view will contain one row for each
+ combination of backend type, target I/O object, and I/O context, showing
+ cluster-wide I/O statistics. Combinations which do not make sense are
+ omitted.
+ </para>
+
+ <para>
+ Currently, I/O on relations (e.g. tables, indexes) is tracked. However,
+ relation I/O which bypasses shared buffers (e.g. when moving a table from one
+ tablespace to another) is currently not tracked.
+ </para>
+
+ <table id="pg-stat-io-view" xreflabel="pg_stat_io">
+ <title><structname>pg_stat_io</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>backend_type</structfield> <type>text</type>
+ </para>
+ <para>
+ Type of backend (e.g. background worker, autovacuum worker). See <link
+ linkend="monitoring-pg-stat-activity-view">
+ <structname>pg_stat_activity</structname></link> for more information
+ on <varname>backend_type</varname>s. Some
+ <varname>backend_type</varname>s do not accumulate I/O operation
+ statistics and will not be included in the view.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_object</structfield> <type>text</type>
+ </para>
+ <para>
+ Target object of an I/O operation. Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>relation</literal>: Permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>temp relation</literal>: Temporary relations.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_context</structfield> <type>text</type>
+ </para>
+ <para>
+ The context of an I/O operation. Possible values are:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>normal</literal>: The default or standard
+ <varname>io_context</varname> for a type of I/O operation. For
+ example, by default, relation data is read into and written out from
+ shared buffers. Thus, reads and writes of relation data to and from
+ shared buffers are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>vacuum</literal>: I/O operations performed outside of shared
+ buffers while vacuuming and analyzing permanent relations. Temporary
+ table vacuums use the same local buffer pool as other temporary table
+ IO operations and are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkread</literal>: Certain large read I/O operations
+ done outside of shared buffers, for example, a sequential scan of a
+ large table.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkwrite</literal>: Certain large write I/O operations
+ done outside of shared buffers, such as <command>COPY</command>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reads</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of read operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>writes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of write operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>extends</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of relation extend operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>op_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of bytes per unit of I/O read, written, or extended.
+ </para>
+ <para>
+ Relation data reads, writes, and extends are done in
+ <varname>block_size</varname> units, derived from the build-time
+ parameter <symbol>BLCKSZ</symbol>, which is <literal>8192</literal> by
+ default.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a block has been written out from a shared or local
+ buffer in order to make it available for another use.
+ </para>
+ <para>
+ In <varname>io_context</varname> <literal>normal</literal>, this counts
+ the number of times a block was evicted from a buffer and replaced with
+ another block. In <varname>io_context</varname>s
+ <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
+ <literal>vacuum</literal>, this counts the number of times a block was
+ evicted from shared buffers in order to add the shared buffer to a
+ separate, size-limited ring buffer for use in a bulk I/O operation.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reuses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of times an existing buffer in a size-limited ring buffer
+ outside of shared buffers was reused as part of an I/O operation in the
+ <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
+ <literal>vacuum</literal> <varname>io_context</varname>s.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>fsyncs</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of <literal>fsync</literal> calls. These are only tracked in
+ <varname>io_context</varname> <literal>normal</literal>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset.
+ </para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Some backend types never perform I/O operations on some I/O objects and/or
+ in some I/O contexts. These rows are omitted from the view. For example, the
+ checkpointer does not checkpoint temporary tables, so there will be no rows
+ for <varname>backend_type</varname> <literal>checkpointer</literal> and
+ <varname>io_object</varname> <literal>temp relation</literal>.
+ </para>
+
+ <para>
+ In addition, some I/O operations will never be performed either by certain
+ backend types or on certain I/O objects and/or in certain I/O contexts.
+ These cells will be NULL. For example, temporary tables are not
+ <literal>fsync</literal>ed, so <varname>fsyncs</varname> will be NULL for
+ <varname>io_object</varname> <literal>temp relation</literal>. Also, the
+ background writer does not perform reads, so <varname>reads</varname> will
+ be NULL in rows for <varname>backend_type</varname> <literal>background
+ writer</literal>.
+ </para>
+
+ <para>
+ <structname>pg_stat_io</structname> can be used to inform database tuning.
+ For example:
+ <itemizedlist>
+ <listitem>
+ <para>
+ A high <varname>evictions</varname> count can indicate that shared
+ buffers should be increased.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Client backends rely on the checkpointer to ensure data is persisted to
+ permanent storage. Large numbers of <varname>fsyncs</varname> by
+ <literal>client backend</literal>s could indicate a misconfiguration of
+ shared buffers or of the checkpointer. More information on configuring
+ the checkpointer can be found in <xref linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Normally, client backends should be able to rely on auxiliary processes
+ like the checkpointer and the background writer to write out dirty data
+ as much as possible. Large numbers of writes by client backends could
+ indicate a misconfiguration of shared buffers or of the checkpointer.
+ More information on configuring the checkpointer can be found in <xref
+ linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
</sect2>
--
2.34.1
[text/x-patch] v49-0002-pgstat-Count-IO-for-relations.patch (22.2K, ../../CAAKRu_YJpZ_fV_n5PdOLWp9NcDZP2n=OZyLSUQRwBCz0iYpK_g@mail.gmail.com/4-v49-0002-pgstat-Count-IO-for-relations.patch)
download | inline diff:
From cb2dd852c8435537ed9a9a148c719e81c0dc22ce Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 17 Jan 2023 16:25:31 -0500
Subject: [PATCH v49 2/4] pgstat: Count IO for relations
Count IOOps done on IOObjects in IOContexts by various BackendTypes
using the IO stats infrastructure introduced by a previous commit.
The primary concern of these statistics is IO operations on data blocks
during the course of normal database operations. IO operations done by,
for example, the archiver or syslogger are not counted in these
statistics. WAL IO, temporary file IO, and IO done directly though smgr*
functions (such as when building an index) are not yet counted but would
be useful future additions.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
src/backend/storage/buffer/bufmgr.c | 111 ++++++++++++++++++++++----
src/backend/storage/buffer/freelist.c | 58 ++++++++++----
src/backend/storage/buffer/localbuf.c | 13 ++-
src/backend/storage/smgr/md.c | 24 ++++++
src/include/storage/buf_internals.h | 8 +-
src/include/storage/bufmgr.h | 7 +-
6 files changed, 185 insertions(+), 36 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8075828e8a..ff12bc2ba6 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -481,8 +481,9 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr);
-static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
+ bool *foundPtr, IOContext *io_context);
+static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
+ IOObject io_object, IOContext io_context);
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -823,6 +824,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BufferDesc *bufHdr;
Block bufBlock;
bool found;
+ IOContext io_context;
+ IOObject io_object;
bool isExtend;
bool isLocalBuf = SmgrIsTemp(smgr);
@@ -855,7 +858,14 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isLocalBuf)
{
- bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found);
+ /*
+ * LocalBufferAlloc() will set the io_context to IOCONTEXT_NORMAL. We
+ * do not use a BufferAccessStrategy for I/O of temporary tables.
+ * However, in some cases, the "strategy" may not be NULL, so we can't
+ * rely on IOContextForStrategy() to set the right IOContext for us.
+ * This may happen in cases like CREATE TEMPORARY TABLE AS...
+ */
+ bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found, &io_context);
if (found)
pgBufferUsage.local_blks_hit++;
else if (isExtend)
@@ -871,7 +881,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, &found, &io_context);
if (found)
pgBufferUsage.shared_blks_hit++;
else if (isExtend)
@@ -986,7 +996,16 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
Assert(!(pg_atomic_read_u32(&bufHdr->state) & BM_VALID)); /* spinlock not needed */
- bufBlock = isLocalBuf ? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr);
+ if (isLocalBuf)
+ {
+ bufBlock = LocalBufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ bufBlock = BufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_RELATION;
+ }
if (isExtend)
{
@@ -995,6 +1014,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
/* don't set checksum for all-zero page */
smgrextend(smgr, forkNum, blockNum, (char *) bufBlock, false);
+ pgstat_count_io_op(io_object, io_context, IOOP_EXTEND);
+
/*
* NB: we're *not* doing a ScheduleBufferTagForWriteback here;
* although we're essentially performing a write. At least on linux
@@ -1020,6 +1041,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgrread(smgr, forkNum, blockNum, (char *) bufBlock);
+ pgstat_count_io_op(io_object, io_context, IOOP_READ);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -1113,14 +1136,19 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* *foundPtr is actually redundant with the buffer's BM_VALID flag, but
* we keep it for simplicity in ReadBuffer.
*
+ * io_context is passed as an output parameter to avoid calling
+ * IOContextForStrategy() when there is a shared buffers hit and no IO
+ * statistics need be captured.
+ *
* No locks are held either at entry or exit.
*/
static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
+ bool from_ring;
BufferTag newTag; /* identity of requested block */
uint32 newHash; /* hash value for newTag */
LWLock *newPartitionLock; /* buffer partition lock for it */
@@ -1172,8 +1200,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* If we get here, previous attempts to read the buffer must
- * have failed ... but we shall bravely try again.
+ * have failed ... but we shall bravely try again. Set
+ * io_context since we will in fact need to count an IO
+ * Operation.
*/
+ *io_context = IOContextForStrategy(strategy);
*foundPtr = false;
}
}
@@ -1187,6 +1218,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ *io_context = IOContextForStrategy(strategy);
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1200,7 +1233,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* Select a victim buffer. The buffer is returned with its header
* spinlock still held!
*/
- buf = StrategyGetBuffer(strategy, &buf_state);
+ buf = StrategyGetBuffer(strategy, &buf_state, &from_ring);
Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 0);
@@ -1254,7 +1287,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
if (XLogNeedsFlush(lsn) &&
- StrategyRejectBuffer(strategy, buf))
+ StrategyRejectBuffer(strategy, buf, from_ring))
{
/* Drop lock/pin and loop around for another buffer */
LWLockRelease(BufferDescriptorGetContentLock(buf));
@@ -1269,7 +1302,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgr->smgr_rlocator.locator.dbOid,
smgr->smgr_rlocator.locator.relNumber);
- FlushBuffer(buf, NULL);
+ FlushBuffer(buf, NULL, IOOBJECT_RELATION, *io_context);
LWLockRelease(BufferDescriptorGetContentLock(buf));
ScheduleBufferTagForWriteback(&BackendWritebackContext,
@@ -1450,6 +1483,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
LWLockRelease(newPartitionLock);
+ if (oldFlags & BM_VALID)
+ {
+ /*
+ * When a BufferAccessStrategy is in use, blocks evicted from shared
+ * buffers are counted as IOOP_EVICT in the corresponding context
+ * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
+ * strategy in two cases: 1) while initially claiming buffers for the
+ * strategy ring 2) to replace an existing strategy ring buffer
+ * because it is pinned or in use and cannot be reused.
+ *
+ * Blocks evicted from buffers already in the strategy ring are
+ * counted as IOOP_REUSE in the corresponding strategy context.
+ *
+ * At this point, we can accurately count evictions and reuses,
+ * because we have successfully claimed the valid buffer. Previously,
+ * we may have been forced to release the buffer due to concurrent
+ * pinners or erroring out.
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, *io_context,
+ from_ring ? IOOP_REUSE : IOOP_EVICT);
+ }
+
/*
* Buffer contents are currently invalid. Try to obtain the right to
* start I/O. If StartBufferIO returns false, then someone else managed
@@ -2570,7 +2625,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
@@ -2820,7 +2875,8 @@ BufferGetTag(Buffer buffer, RelFileLocator *rlocator, ForkNumber *forknum,
* as the second parameter. If not, pass NULL.
*/
static void
-FlushBuffer(BufferDesc *buf, SMgrRelation reln)
+FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
+ IOContext io_context)
{
XLogRecPtr recptr;
ErrorContextCallback errcallback;
@@ -2912,6 +2968,26 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
bufToWrite,
false);
+ /*
+ * When a strategy is in use, only flushes of dirty buffers already in the
+ * strategy ring are counted as strategy writes (IOCONTEXT
+ * [BULKREAD|BULKWRITE|VACUUM] IOOP_WRITE) for the purpose of IO
+ * statistics tracking.
+ *
+ * If a shared buffer initially added to the ring must be flushed before
+ * being used, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE.
+ *
+ * If a shared buffer which was added to the ring later because the
+ * current strategy buffer is pinned or in use or because all strategy
+ * buffers were dirty and rejected (for BAS_BULKREAD operations only)
+ * requires flushing, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE
+ * (from_ring will be false).
+ *
+ * When a strategy is not in use, the write can only be a "regular" write
+ * of a dirty shared buffer (IOCONTEXT_NORMAL IOOP_WRITE).
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, io_context, IOOP_WRITE);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -3554,6 +3630,8 @@ FlushRelationBuffers(Relation rel)
buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_WRITE);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -3586,7 +3664,8 @@ FlushRelationBuffers(Relation rel)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, RelationGetSmgr(rel));
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOCONTEXT_NORMAL, IOOBJECT_RELATION);
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3684,7 +3763,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, srelent->srel);
+ FlushBuffer(bufHdr, srelent->srel, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3894,7 +3973,7 @@ FlushDatabaseBuffers(Oid dbid)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3921,7 +4000,7 @@ FlushOneBuffer(Buffer buffer)
Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr)));
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7dec35801c..c690d5f15f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,7 @@
*/
#include "postgres.h"
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -81,12 +82,6 @@ typedef struct BufferAccessStrategyData
*/
int current;
- /*
- * True if the buffer just returned by StrategyGetBuffer had been in the
- * ring already.
- */
- bool current_was_in_ring;
-
/*
* Array of buffer numbers. InvalidBuffer (that is, zero) indicates we
* have not yet selected a buffer for this ring slot. For allocation
@@ -198,13 +193,15 @@ have_free_buffer(void)
* return the buffer with the buffer header spinlock still held.
*/
BufferDesc *
-StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
+StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring)
{
BufferDesc *buf;
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ *from_ring = false;
+
/*
* If given a strategy object, see whether it can select a buffer. We
* assume strategy objects don't need buffer_strategy_lock.
@@ -213,7 +210,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
{
buf = GetBufferFromRing(strategy, buf_state);
if (buf != NULL)
+ {
+ *from_ring = true;
return buf;
+ }
}
/*
@@ -602,7 +602,7 @@ FreeAccessStrategy(BufferAccessStrategy strategy)
/*
* GetBufferFromRing -- returns a buffer from the ring, or NULL if the
- * ring is empty.
+ * ring is empty / not usable.
*
* The bufhdr spin lock is held on the returned buffer.
*/
@@ -625,10 +625,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
*/
bufnum = strategy->buffers[strategy->current];
if (bufnum == InvalidBuffer)
- {
- strategy->current_was_in_ring = false;
return NULL;
- }
/*
* If the buffer is pinned we cannot use it under any circumstances.
@@ -644,7 +641,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) <= 1)
{
- strategy->current_was_in_ring = true;
*buf_state = local_buf_state;
return buf;
}
@@ -654,7 +650,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
* Tell caller to allocate a new buffer with the normal allocation
* strategy. He'll then replace this ring element via AddBufferToRing.
*/
- strategy->current_was_in_ring = false;
return NULL;
}
@@ -670,6 +665,39 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
strategy->buffers[strategy->current] = BufferDescriptorGetBuffer(buf);
}
+/*
+ * Utility function returning the IOContext of a given BufferAccessStrategy's
+ * strategy ring.
+ */
+IOContext
+IOContextForStrategy(BufferAccessStrategy strategy)
+{
+ if (!strategy)
+ return IOCONTEXT_NORMAL;
+
+ switch (strategy->btype)
+ {
+ case BAS_NORMAL:
+
+ /*
+ * Currently, GetAccessStrategy() returns NULL for
+ * BufferAccessStrategyType BAS_NORMAL, so this case is
+ * unreachable.
+ */
+ pg_unreachable();
+ return IOCONTEXT_NORMAL;
+ case BAS_BULKREAD:
+ return IOCONTEXT_BULKREAD;
+ case BAS_BULKWRITE:
+ return IOCONTEXT_BULKWRITE;
+ case BAS_VACUUM:
+ return IOCONTEXT_VACUUM;
+ }
+
+ elog(ERROR, "unrecognized BufferAccessStrategyType: %d", strategy->btype);
+ pg_unreachable();
+}
+
/*
* StrategyRejectBuffer -- consider rejecting a dirty buffer
*
@@ -682,14 +710,14 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
* if this buffer should be written and re-used.
*/
bool
-StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
+StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring)
{
/* We only do this in bulkread mode */
if (strategy->btype != BAS_BULKREAD)
return false;
/* Don't muck with behavior of normal buffer-replacement strategy */
- if (!strategy->current_was_in_ring ||
+ if (!from_ring ||
strategy->buffers[strategy->current] != BufferDescriptorGetBuffer(buf))
return false;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8372acc383..8e286db5df 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "catalog/catalog.h"
#include "executor/instrument.h"
+#include "pgstat.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "utils/guc_hooks.h"
@@ -107,7 +108,7 @@ PrefetchLocalBuffer(SMgrRelation smgr, ForkNumber forkNum,
*/
BufferDesc *
LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
BufferTag newTag; /* identity of requested block */
LocalBufferLookupEnt *hresult;
@@ -127,6 +128,14 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
+ /*
+ * IO Operations on local buffers are only done in IOCONTEXT_NORMAL. Set
+ * io_context here (instead of after a buffer hit would have returned) for
+ * convenience since we don't have to worry about the overhead of calling
+ * IOContextForStrategy().
+ */
+ *io_context = IOCONTEXT_NORMAL;
+
if (hresult)
{
b = hresult->id;
@@ -230,6 +239,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
buf_state &= ~BM_DIRTY;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_WRITE);
pgBufferUsage.local_blks_written++;
}
@@ -256,6 +266,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
ClearBufferTag(&bufHdr->tag);
buf_state &= ~(BM_VALID | BM_TAG_VALID);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_EVICT);
}
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 60c9905eff..8da813600c 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -983,6 +983,15 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
{
MdfdVec *v = &reln->md_seg_fds[forknum][segno - 1];
+ /*
+ * fsyncs done through mdimmedsync() should be tracked in a separate
+ * IOContext than those done through mdsyncfiletag() to differentiate
+ * between unavoidable client backend fsyncs (e.g. those done during
+ * index build) and those which ideally would have been done by the
+ * checkpointer. Since other IO operations bypassing the buffer
+ * manager could also be tracked in such an IOContext, wait until
+ * these are also tracked to track immediate fsyncs.
+ */
if (FileSync(v->mdfd_vfd, WAIT_EVENT_DATA_FILE_IMMEDIATE_SYNC) < 0)
ereport(data_sync_elevel(ERROR),
(errcode_for_file_access(),
@@ -1021,6 +1030,19 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
{
+ /*
+ * We have no way of knowing if the current IOContext is
+ * IOCONTEXT_NORMAL or IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] at this
+ * point, so count the fsync as being in the IOCONTEXT_NORMAL
+ * IOContext. This is probably okay, because the number of backend
+ * fsyncs doesn't say anything about the efficacy of the
+ * BufferAccessStrategy. And counting both fsyncs done in
+ * IOCONTEXT_NORMAL and IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] under
+ * IOCONTEXT_NORMAL is likely clearer when investigating the number of
+ * backend fsyncs.
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL, IOOP_FSYNC);
+
ereport(DEBUG1,
(errmsg_internal("could not forward fsync request because request queue is full")));
@@ -1410,6 +1432,8 @@ mdsyncfiletag(const FileTag *ftag, char *path)
if (need_to_close)
FileClose(file);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL, IOOP_FSYNC);
+
errno = save_errno;
return result;
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index ed8aa2519c..0b44814740 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -15,6 +15,7 @@
#ifndef BUFMGR_INTERNALS_H
#define BUFMGR_INTERNALS_H
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
@@ -391,11 +392,12 @@ extern void IssuePendingWritebacks(WritebackContext *context);
extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *tag);
/* freelist.c */
+extern IOContext IOContextForStrategy(BufferAccessStrategy bas);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
- uint32 *buf_state);
+ uint32 *buf_state, bool *from_ring);
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
- BufferDesc *buf);
+ BufferDesc *buf, bool from_ring);
extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
extern void StrategyNotifyBgWriter(int bgwprocno);
@@ -417,7 +419,7 @@ extern PrefetchBufferResult PrefetchLocalBuffer(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum);
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
- BlockNumber blockNum, bool *foundPtr);
+ BlockNumber blockNum, bool *foundPtr, IOContext *io_context);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 33eadbc129..b8a18b8081 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -23,7 +23,12 @@
typedef void *Block;
-/* Possible arguments for GetAccessStrategy() */
+/*
+ * Possible arguments for GetAccessStrategy().
+ *
+ * If adding a new BufferAccessStrategyType, also add a new IOContext so
+ * IO statistics using this strategy are tracked.
+ */
typedef enum BufferAccessStrategyType
{
BAS_NORMAL, /* Normal random access */
--
2.34.1
[text/x-patch] v49-0003-Add-system-view-tracking-IO-ops-per-backend-type.patch (32.6K, ../../CAAKRu_YJpZ_fV_n5PdOLWp9NcDZP2n=OZyLSUQRwBCz0iYpK_g@mail.gmail.com/5-v49-0003-Add-system-view-tracking-IO-ops-per-backend-type.patch)
download | inline diff:
From d40934679b00fd1e157bd6942d7f3faf8be5ea8e Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 17 Jan 2023 16:28:27 -0500
Subject: [PATCH v49 3/4] Add system view tracking IO ops per backend type
Add pg_stat_io, a system view which tracks the number of IOOps
(evictions, reuses, reads, writes, extensions, and fsyncs) done on each
IOObject (relation, temp relation) in each IOContext ("normal" and those
using a BufferAccessStrategy) by each type of backend (e.g. client
backend, checkpointer).
Some BackendTypes do not accumulate IO operations statistics and will
not be included in the view.
Some IOObjects are never operated on in some IOContexts or by some
BackendTypes. These rows are omitted from the view. For example,
checkpointer will never operate on IOOBJECT_TEMP_RELATION data, so those
rows are omitted.
Some IOContexts are not used by some BackendTypes and will not be in the
view. For example, checkpointer does not use a BufferAccessStrategy
(currently), so there will be no rows for BufferAccessStrategy
IOContexts for checkpointer.
Some IOOps are invalid in combination with certain IOContexts and
certain IOObjects. Those cells will be NULL in the view to distinguish
between 0 observed IOOps of that type and an invalid combination. For
example, temporary tables are not fsynced so cells for all BackendTypes
for IOOBJECT_TEMP_RELATION and IOOP_FSYNC will be NULL.
Some BackendTypes never perform certain IOOps. Those cells will also be
NULL in the view. For example, bgwriter should not perform reads.
View stats are populated with statistics incremented when a backend
performs an IO Operation and maintained by the cumulative statistics
subsystem.
Each row of the view shows stats for a particular BackendType, IOObject,
IOContext combination (e.g. a client backend's operations on permanent
relations in shared buffers) and each column in the view is the total
number of IO Operations done (e.g. writes). So a cell in the view would
be, for example, the number of blocks of relation data written from
shared buffers by client backends since the last stats reset.
In anticipation of tracking WAL IO and non-block-oriented IO (such as
temporary file IO), the "op_bytes" column specifies the unit of the
"reads", "writes", and "extends" columns for a given row.
Note that some of the cells in the view are redundant with fields in
pg_stat_bgwriter (e.g. buffers_backend), however these have been kept in
pg_stat_bgwriter for backwards compatibility. Deriving the redundant
pg_stat_bgwriter stats from the IO operations stats structures was also
problematic due to the separate reset targets for 'bgwriter' and 'io'.
Suggested by Andres Freund
Catalog version should be bumped.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
contrib/amcheck/expected/check_heap.out | 34 ++++
contrib/amcheck/sql/check_heap.sql | 27 +++
src/backend/catalog/system_views.sql | 15 ++
src/backend/utils/adt/pgstatfuncs.c | 141 ++++++++++++++
src/include/catalog/pg_proc.dat | 9 +
src/test/regress/expected/rules.out | 12 ++
src/test/regress/expected/stats.out | 234 ++++++++++++++++++++++++
src/test/regress/sql/stats.sql | 148 +++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 621 insertions(+)
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index c010361025..e4785141a6 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -66,6 +66,22 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -88,6 +104,23 @@ SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock :=
-------+--------+--------+-----
(0 rows)
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+ ?column?
+----------
+ t
+(1 row)
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
SET ROLE regress_heaptest_role;
@@ -195,6 +228,7 @@ ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 298de6886a..6794ca4eb0 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -20,11 +20,29 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
+
-- Add some data so subsequent tests are not entirely trivial
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
+
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -32,6 +50,14 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');
SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
@@ -110,6 +136,7 @@ SELECT * FROM verify_heapam('test_foreign_table',
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d2a8c82900..70699f4b85 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1116,6 +1116,21 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+CREATE VIEW pg_stat_io AS
+SELECT
+ b.backend_type,
+ b.io_object,
+ b.io_context,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+FROM pg_stat_get_io() b;
+
CREATE VIEW pg_stat_wal AS
SELECT
w.wal_records,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 6df9f06a20..5b79d703b7 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1234,6 +1234,147 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+* When adding a new column to the pg_stat_io view, add a new enum value
+* here above IO_NUM_COLUMNS.
+*/
+typedef enum io_stat_col
+{
+ IO_COL_BACKEND_TYPE,
+ IO_COL_IO_OBJECT,
+ IO_COL_IO_CONTEXT,
+ IO_COL_READS,
+ IO_COL_WRITES,
+ IO_COL_EXTENDS,
+ IO_COL_CONVERSION,
+ IO_COL_EVICTIONS,
+ IO_COL_REUSES,
+ IO_COL_FSYNCS,
+ IO_COL_RESET_TIME,
+ IO_NUM_COLUMNS,
+} io_stat_col;
+
+/*
+ * When adding a new IOOp, add a new io_stat_col and add a case to this
+ * function returning the corresponding io_stat_col.
+ */
+static io_stat_col
+pgstat_get_io_op_index(IOOp io_op)
+{
+ switch (io_op)
+ {
+ case IOOP_EVICT:
+ return IO_COL_EVICTIONS;
+ case IOOP_READ:
+ return IO_COL_READS;
+ case IOOP_REUSE:
+ return IO_COL_REUSES;
+ case IOOP_WRITE:
+ return IO_COL_WRITES;
+ case IOOP_EXTEND:
+ return IO_COL_EXTENDS;
+ case IOOP_FSYNC:
+ return IO_COL_FSYNCS;
+ }
+
+ elog(ERROR, "unrecognized IOOp value: %d", io_op);
+ pg_unreachable();
+}
+
+Datum
+pg_stat_get_io(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo;
+ PgStat_IO *backends_io_stats;
+ Datum reset_time;
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ backends_io_stats = pgstat_fetch_stat_io();
+
+ reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
+
+ for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ {
+ Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
+ PgStat_BktypeIO *bktype_stats = &backends_io_stats->stats[bktype];
+
+ /*
+ * In Assert builds, we can afford an extra loop through all of the
+ * counters checking that only expected stats are non-zero, since it
+ * keeps the non-Assert code cleaner.
+ */
+ Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype));
+
+ /*
+ * For those BackendTypes without IO Operation stats, skip
+ * representing them in the view altogether.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ continue;
+
+ for (IOObject io_obj = IOOBJECT_FIRST;
+ io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ {
+ const char *obj_name = pgstat_get_io_object_name(io_obj);
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ const char *context_name = pgstat_get_io_context_name(io_context);
+
+ Datum values[IO_NUM_COLUMNS] = {0};
+ bool nulls[IO_NUM_COLUMNS] = {0};
+
+ /*
+ * Some combinations of BackendType, IOObject, and IOContext
+ * are not valid for any type of IOOp. In such cases, omit the
+ * entire row from the view.
+ */
+ if (!pgstat_tracks_io_object(bktype, io_obj, io_context))
+ continue;
+
+ values[IO_COL_BACKEND_TYPE] = bktype_desc;
+ values[IO_COL_IO_CONTEXT] = CStringGetTextDatum(context_name);
+ values[IO_COL_IO_OBJECT] = CStringGetTextDatum(obj_name);
+ values[IO_COL_RESET_TIME] = TimestampTzGetDatum(reset_time);
+
+ /*
+ * Hard-code this to the value of BLCKSZ for now. Future
+ * values could include XLOG_BLCKSZ, once WAL IO is tracked,
+ * and constant multipliers, once non-block-oriented IO (e.g.
+ * temporary file IO) is tracked.
+ */
+ values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
+
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ int col_idx = pgstat_get_io_op_index(io_op);
+
+ /*
+ * Some combinations of BackendType and IOOp, of IOContext
+ * and IOOp, and of IOObject and IOOp are not tracked. Set
+ * these cells in the view NULL.
+ */
+ nulls[col_idx] = !pgstat_tracks_io_op(bktype, io_obj, io_context, io_op);
+
+ if (nulls[col_idx])
+ continue;
+
+ values[col_idx] =
+ Int64GetDatum(bktype_stats->data[io_obj][io_context][io_op]);
+ }
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ }
+ }
+
+ return (Datum) 0;
+}
+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3810de7b22..2155d93b44 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5690,6 +5690,15 @@
proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
+{ oid => '8459', descr => 'statistics: per backend type IO statistics',
+ proname => 'pg_stat_get_io', provolatile => 'v',
+ prorows => '30', proretset => 't',
+ proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,text,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{backend_type,io_object,io_context,reads,writes,extends,op_bytes,evictions,reuses,fsyncs,stats_reset}',
+ prosrc => 'pg_stat_get_io' },
+
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index a969ae63eb..8a7ed673c2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1876,6 +1876,18 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_enc AS encrypted
FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
WHERE (s.client_port IS NOT NULL);
+pg_stat_io| SELECT b.backend_type,
+ b.io_object,
+ b.io_context,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+ FROM pg_stat_get_io() b(backend_type, io_object, io_context, reads, writes, extends, op_bytes, evictions, reuses, fsyncs, stats_reset);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 1d84407a03..46bc79e740 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1126,4 +1126,238 @@ SELECT pg_stat_get_subscription_stats(NULL);
(1 row)
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+ ?column?
+----------
+ t
+(1 row)
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+ count
+-------
+ 100
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+ count
+-------
+ 5000
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+ ?column? | ?column? | ?column? | ?column?
+----------+----------+----------+----------
+ t | t | t | t
+(1 row)
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
+
+RESET wal_skip_threshold;
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+ pg_stat_have_stats
+--------------------
+ t
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+ ?column?
+----------
+ t
+(1 row)
+
-- End of Stats Test
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index b4d6753c71..4465649211 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -536,4 +536,152 @@ SELECT pg_stat_get_replication_slot(NULL);
SELECT pg_stat_get_subscription_stats(NULL);
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_io_stats_tblspc;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_shared;
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+SELECT pg_stat_force_next_flush();
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_io_stats_tblspc;
+SELECT pg_stat_force_next_flush();
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+-- Drop the table so we can drop the tablespace later.
+DROP TABLE test_io_local;
+RESET temp_buffers;
+DROP TABLESPACE regress_io_stats_tblspc;
+
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+RESET wal_skip_threshold;
+
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+
-- End of Stats Test
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1be6e07980..a399e0a5e4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3377,6 +3377,7 @@ intset_internal_node
intset_leaf_node
intset_node
intvKEY
+io_stat_col
itemIdCompact
itemIdCompactData
iterator
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-19 11:17 ` vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: vignesh C @ 2023-01-19 11:17 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Andres Freund <[email protected]>; Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>
On Wed, 18 Jan 2023 at 03:30, Melanie Plageman
<[email protected]> wrote:
>
> v49 attached
>
> On Tue, Jan 17, 2023 at 2:12 PM Andres Freund <[email protected]> wrote:
> > On 2023-01-17 12:22:14 -0500, Melanie Plageman wrote:
> >
> > > > > +typedef struct PgStat_BackendIO
> > > > > +{
> > > > > + PgStat_Counter data[IOCONTEXT_NUM_TYPES][IOOBJECT_NUM_TYPES][IOOP_NUM_TYPES];
> > > > > +} PgStat_BackendIO;
> > > >
> > > > Would it bother you if we swapped the order of iocontext and iobject here and
> > > > related places? It makes more sense to me semantically, and should now be
> > > > pretty easy, code wise.
> > >
> > > So, thinking about this I started noticing inconsistencies in other
> > > areas around this order:
> > > For example: ordering of objects mentioned in commit messages and comments,
> > > ordering of parameters (like in pgstat_count_io_op() [currently in
> > > reverse order]).
> > >
> > > I think we should make a final decision about this ordering and then
> > > make everywhere consistent (including ordering in the view).
> > >
> > > Currently the order is:
> > > BackendType
> > > IOContext
> > > IOObject
> > > IOOp
> > >
> > > You are suggesting this order:
> > > BackendType
> > > IOObject
> > > IOContext
> > > IOOp
> > >
> > > Could you explain what you find more natural about this ordering (as I
> > > find the other more natural)?
> >
> > The object we're performing IO on determines more things than the context. So
> > it just seems like the natural hierarchical fit. The context is a sub-category
> > of the object. Consider how it'll look like if we also have objects for 'wal',
> > 'temp files'. It'll make sense to group by just the object, but it won't make
> > sense to group by just the context.
> >
> > If it were trivial to do I'd use a different IOContext for each IOObject. But
> > it'd make it much harder. So there'll just be a bunch of values of IOContext
> > that'll only be used for one or a subset of the IOObjects.
> >
> >
> > The reason to put BackendType at the top is pragmatic - one backend is of a
> > single type, but can do IO for all kinds of objects/contexts. So any other
> > hierarchy would make the locking etc much harder.
> >
> >
> > > This is one possible natural sentence with these objects:
> > >
> > > During COPY, a client backend may read in data from a permanent
> > > relation.
> > > This order is:
> > > IOContext
> > > BackendType
> > > IOOp
> > > IOObject
> > >
> > > I think English sentences are often structured subject, verb, object --
> > > but in our case, we have an extra thing that doesn't fit neatly
> > > (IOContext).
> >
> > "..., to avoid polluting the buffer cache it uses the bulk (read|write)
> > strategy".
> >
> >
> > > Also, IOOp in a sentence would be in the middle (as the
> > > verb). I made it last because a) it feels like the smallest unit b) it
> > > would make the code a lot more annoying if it wasn't last.
> >
> > Yea, I think pragmatically that is the right choice.
>
> I have changed the order and updated all the places using
> PgStat_BktypeIO as well as in all locations in which it should be
> ordered for consistency (that I could find in the pass I did) -- e.g.
> the view definition, function signatures, comments, commit messages,
> etc.
>
> > > > > +-- Change the tablespace so that the table is rewritten directly, then SELECT
> > > > > +-- from it to cause it to be read back into shared buffers.
> > > > > +SET allow_in_place_tablespaces = true;
> > > > > +CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
> > > >
> > > > Perhaps worth doing this in tablespace.sql, to avoid the additional
> > > > checkpoints done as part of CREATE/DROP TABLESPACE?
> > > >
> > > > Or, at least combine this with the CHECKPOINTs above?
> > >
> > > I see a checkpoint is requested when dropping the tablespace if not all
> > > the files in it are deleted. It seems like if the DROP TABLE for the
> > > permanent table is before the explicit checkpoints in the test, then the
> > > DROP TABLESPACE will not cause an additional checkpoint.
> >
> > Unfortunately, that's not how it works :(. See the comment above mdunlink():
> >
> > > * For regular relations, we don't unlink the first segment file of the rel,
> > > * but just truncate it to zero length, and record a request to unlink it after
> > > * the next checkpoint. Additional segments can be unlinked immediately,
> > > * however. Leaving the empty file in place prevents that relfilenumber
> > > * from being reused. The scenario this protects us from is:
> > > ...
> >
> >
> > > Is this what you are suggesting? Dropping the temporary table should not
> > > have an effect on this.
> >
> > I was wondering about simply moving that portion of the test to
> > tablespace.sql, where we already created a tablespace.
> >
> >
> > An alternative would be to propose splitting tablespace.sql into one portion
> > running at the start of parallel_schedule, and one at the end. Historically,
> > we needed tablespace.sql to be optional due to causing problems when
> > replicating to another instance on the same machine, but now we have
> > allow_in_place_tablespaces.
>
> It seems like the best way would be to split up the tablespace test file
> as you suggested and drop the tablespace at the end of the regression
> test suite. There could be other tests that could use a tablespace.
> Though what I wrote is kind of tablespace test coverage, if this
> rewriting behavior no longer happened when doing alter table set
> tablespace, we would want to come up with a new test which exercised
> that code to count those IO stats, not simply delete it from the
> tablespace tests.
>
> > > > SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
> > > >
> > > > Better toast compression or such could easily make test_io_local smaller than
> > > > it's today. Seeing that it's too small would make it easier to understand the
> > > > failure.
> > >
> > > Good idea. So, I used pg_table_size() because it seems like
> > > pg_relation_size() does not include the toast relations. However, I'm
> > > not sure this is a good idea, because pg_table_size() includes FSM and
> > > visibility map. Should I write a query to get the toast relation name
> > > and add pg_relation_size() of that relation and the main relation?
> >
> > I think it's the right thing to just include the relation size. Your queries
> > IIRC won't use the toast table or other forks. So I'd leave it at just
> > pg_relation_size().
>
> I did notice that this test wasn't using the toast table for the
> toastable column -- but you mentioned better toast compression affecting
> the future test stability, so I'm confused.
The patch does not apply on top of HEAD as in [1], please post a rebased patch:
=== Applying patches on top of PostgreSQL commit ID
4f74f5641d53559ec44e74d5bf552e167fdd5d20 ===
=== applying patch
./v49-0003-Add-system-view-tracking-IO-ops-per-backend-type.patch
....
patching file src/test/regress/expected/rules.out
Hunk #1 FAILED at 1876.
1 out of 1 hunk FAILED -- saving rejects to file
src/test/regress/expected/rules.out.rej
[1] - http://cfbot.cputube.org/patch_41_3272.log
Regards,
Vignesh
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
@ 2023-01-19 21:28 ` Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Melanie Plageman @ 2023-01-19 21:28 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Andres Freund <[email protected]>; Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>
On Thu, Jan 19, 2023 at 6:18 AM vignesh C <[email protected]> wrote:
> The patch does not apply on top of HEAD as in [1], please post a rebased patch:
> === Applying patches on top of PostgreSQL commit ID
> 4f74f5641d53559ec44e74d5bf552e167fdd5d20 ===
> === applying patch
> ./v49-0003-Add-system-view-tracking-IO-ops-per-backend-type.patch
> ....
> patching file src/test/regress/expected/rules.out
> Hunk #1 FAILED at 1876.
> 1 out of 1 hunk FAILED -- saving rejects to file
> src/test/regress/expected/rules.out.rej
>
> [1] - http://cfbot.cputube.org/patch_41_3272.log
Yes, it conflicted with 47bb9db75996232. rebased v50 is attached.
On Tue, Jan 17, 2023 at 5:00 PM Melanie Plageman
<[email protected]> wrote:
> > > > > +-- Change the tablespace so that the table is rewritten directly, then SELECT
> > > > > +-- from it to cause it to be read back into shared buffers.
> > > > > +SET allow_in_place_tablespaces = true;
> > > > > +CREATE TABLESPACE regress_io_stats_tblspc LOCATION '';
> > > >
> > > > Perhaps worth doing this in tablespace.sql, to avoid the additional
> > > > checkpoints done as part of CREATE/DROP TABLESPACE?
> > > >
> > > > Or, at least combine this with the CHECKPOINTs above?
> > >
> > > I see a checkpoint is requested when dropping the tablespace if not all
> > > the files in it are deleted. It seems like if the DROP TABLE for the
> > > permanent table is before the explicit checkpoints in the test, then the
> > > DROP TABLESPACE will not cause an additional checkpoint.
> >
> > Unfortunately, that's not how it works :(. See the comment above mdunlink():
> >
> > > * For regular relations, we don't unlink the first segment file of the rel,
> > > * but just truncate it to zero length, and record a request to unlink it after
> > > * the next checkpoint. Additional segments can be unlinked immediately,
> > > * however. Leaving the empty file in place prevents that relfilenumber
> > > * from being reused. The scenario this protects us from is:
> > > ...
> >
> >
> > > Is this what you are suggesting? Dropping the temporary table should not
> > > have an effect on this.
> >
> > I was wondering about simply moving that portion of the test to
> > tablespace.sql, where we already created a tablespace.
> >
> >
> > An alternative would be to propose splitting tablespace.sql into one portion
> > running at the start of parallel_schedule, and one at the end. Historically,
> > we needed tablespace.sql to be optional due to causing problems when
> > replicating to another instance on the same machine, but now we have
> > allow_in_place_tablespaces.
>
> It seems like the best way would be to split up the tablespace test file
> as you suggested and drop the tablespace at the end of the regression
> test suite. There could be other tests that could use a tablespace.
> Though what I wrote is kind of tablespace test coverage, if this
> rewriting behavior no longer happened when doing alter table set
> tablespace, we would want to come up with a new test which exercised
> that code to count those IO stats, not simply delete it from the
> tablespace tests.
I have added a patch to the set which creates the regress_tblspace
(formerly created in tablespace.sq1) in test_setup.sql. I then moved the
tablespace test to the end of the parallel schedule so that my test (and
others) could use the regress_tblspace.
I modified some of the tablespace.sql tests to be more specific in terms
of the objects they are looking for so that tests using the tablespace
are not forced to drop all of the objects they make in the tablespace.
Note that I did not proactively change all tests in tablespace.sql that
may fail in this way -- only those that failed because of the tables I
created (and did not drop) from regress_tblspace.
- Melanie
Attachments:
[text/x-patch] v50-0001-Create-regress_tblspc-in-test_setup.patch (10.2K, ../../CAAKRu_b=1cSOq6Ln96GYXaeMfUnVzpjSBXWvi_RNBJbQ0WnXug@mail.gmail.com/2-v50-0001-Create-regress_tblspc-in-test_setup.patch)
download | inline diff:
From 3976128fab1467fde8ee7e1bc1f54b023e96e35d Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Thu, 19 Jan 2023 15:45:54 -0500
Subject: [PATCH v50 1/5] Create regress_tblspc in test_setup
Other tests may want to use a tablespace. Now that we have
allow_in_place_tablespaces, move the tablespace test to the end of the
parallel schedule and create the main tablespace it uses in test setup.
Modify the tablespace tests a bit to check for specific relations in the
test and not simply for the absence or presence of any objects in the
tablespace in case other tests leave objects around in the tablespace.
---
src/test/regress/expected/tablespace.out | 63 +++++++++++++-----------
src/test/regress/expected/test_setup.out | 3 ++
src/test/regress/parallel_schedule | 9 ++--
src/test/regress/sql/tablespace.sql | 40 ++++++++++-----
src/test/regress/sql/test_setup.sql | 4 ++
5 files changed, 73 insertions(+), 46 deletions(-)
diff --git a/src/test/regress/expected/tablespace.out b/src/test/regress/expected/tablespace.out
index c52cf1cfcf..a2aa95bd97 100644
--- a/src/test/regress/expected/tablespace.out
+++ b/src/test/regress/expected/tablespace.out
@@ -22,8 +22,6 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
--- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '';
-- This returns a relative path as of an effect of allow_in_place_tablespaces,
-- masking the tablespace OID used in the path name.
SELECT regexp_replace(pg_tablespace_location(oid), '(pg_tblspc)/(\d+)', '\1/NNN')
@@ -83,11 +81,14 @@ REINDEX (TABLESPACE regress_tblspace) INDEX regress_tblspace_test_tbl_idx;
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
ROLLBACK;
-- no relation moved to the new tablespace
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace';
- relname
----------
-(0 rows)
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
-- check that all indexes are moved to a new tablespace with different
-- relfilenode.
@@ -102,40 +103,46 @@ SELECT relfilenode as toast_filenode FROM pg_class
WHERE i.indrelid = c.reltoastrelid AND
c.relname = 'regress_tblspace_test_tbl') \gset
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
- relname
--------------------------------
- regress_tblspace_test_tbl_idx
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+ ?column?
+----------
+ t
(1 row)
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE regress_tblspace;
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
- relname
--------------------------------
- regress_tblspace_test_tbl_idx
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+ ?column?
+----------
+ t
(1 row)
-- Move back to the default tablespace.
ALTER INDEX regress_tblspace_test_tbl_idx SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
ORDER BY c.relname;
- relname
----------
-(0 rows)
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
REINDEX (TABLESPACE regress_tblspace, CONCURRENTLY) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
- relname
--------------------------------
- regress_tblspace_test_tbl_idx
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+ ?column?
+----------
+ t
(1 row)
SELECT relfilenode = :main_filenode AS main_same FROM pg_class
diff --git a/src/test/regress/expected/test_setup.out b/src/test/regress/expected/test_setup.out
index 391b36d131..4f54fe20ec 100644
--- a/src/test/regress/expected/test_setup.out
+++ b/src/test/regress/expected/test_setup.out
@@ -18,6 +18,9 @@ SET synchronous_commit = on;
-- and most of the core regression tests still expect that.
--
GRANT ALL ON SCHEMA public TO public;
+-- Create a tablespace we can use in tests.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_tblspace LOCATION '';
--
-- These tables have traditionally been referenced by many tests,
-- so create and populate them. Insert only non-error values here.
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index a930dfe48c..15e015b3d6 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -11,11 +11,6 @@
# required setup steps
test: test_setup
-# run tablespace by itself, and early, because it forces a checkpoint;
-# we'd prefer not to have checkpoints later in the tests because that
-# interferes with crash-recovery testing.
-test: tablespace
-
# ----------
# The first group of parallel tests
# ----------
@@ -132,3 +127,7 @@ test: event_trigger oidjoins
# this test also uses event triggers, so likewise run it by itself
test: fast_default
+
+# run tablespace test at the end because it drops the tablespace created during
+# setup that other tests may use.
+test: tablespace
diff --git a/src/test/regress/sql/tablespace.sql b/src/test/regress/sql/tablespace.sql
index 21db433f2a..1e03d679b2 100644
--- a/src/test/regress/sql/tablespace.sql
+++ b/src/test/regress/sql/tablespace.sql
@@ -20,8 +20,6 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
--- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '';
-- This returns a relative path as of an effect of allow_in_place_tablespaces,
-- masking the tablespace OID used in the path name.
SELECT regexp_replace(pg_tablespace_location(oid), '(pg_tblspc)/(\d+)', '\1/NNN')
@@ -66,7 +64,9 @@ REINDEX (TABLESPACE regress_tblspace) INDEX regress_tblspace_test_tbl_idx;
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
ROLLBACK;
-- no relation moved to the new tablespace
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace';
-- check that all indexes are moved to a new tablespace with different
@@ -74,6 +74,7 @@ SELECT c.relname FROM pg_class c, pg_tablespace s
-- Save first the existing relfilenode for the toast and main relations.
SELECT relfilenode as main_filenode FROM pg_class
WHERE relname = 'regress_tblspace_test_tbl_idx' \gset
+
SELECT relfilenode as toast_filenode FROM pg_class
WHERE oid =
(SELECT i.indexrelid
@@ -81,24 +82,37 @@ SELECT relfilenode as toast_filenode FROM pg_class
pg_index i
WHERE i.indrelid = c.reltoastrelid AND
c.relname = 'regress_tblspace_test_tbl') \gset
+
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
+
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE regress_tblspace;
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
+
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+
-- Move back to the default tablespace.
ALTER INDEX regress_tblspace_test_tbl_idx SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
ORDER BY c.relname;
+
REINDEX (TABLESPACE regress_tblspace, CONCURRENTLY) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
+
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+
SELECT relfilenode = :main_filenode AS main_same FROM pg_class
WHERE relname = 'regress_tblspace_test_tbl_idx';
SELECT relfilenode = :toast_filenode as toast_same FROM pg_class
diff --git a/src/test/regress/sql/test_setup.sql b/src/test/regress/sql/test_setup.sql
index 02c0c84c3a..8439b38d21 100644
--- a/src/test/regress/sql/test_setup.sql
+++ b/src/test/regress/sql/test_setup.sql
@@ -23,6 +23,10 @@ SET synchronous_commit = on;
--
GRANT ALL ON SCHEMA public TO public;
+-- Create a tablespace we can use in tests.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_tblspace LOCATION '';
+
--
-- These tables have traditionally been referenced by many tests,
-- so create and populate them. Insert only non-error values here.
--
2.34.1
[text/x-patch] v50-0002-pgstat-Infrastructure-to-track-IO-operations.patch (28.6K, ../../CAAKRu_b=1cSOq6Ln96GYXaeMfUnVzpjSBXWvi_RNBJbQ0WnXug@mail.gmail.com/3-v50-0002-pgstat-Infrastructure-to-track-IO-operations.patch)
download | inline diff:
From e01f99ceb59fbc85654cf552e4bb527f8076c07a Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 17 Jan 2023 16:10:34 -0500
Subject: [PATCH v50 2/5] pgstat: Infrastructure to track IO operations
Introduce "IOOp", an IO operation done by a backend, "IOObject", the
target object of the IO, and "IOContext", the context or location of the
IO operations on that object. For example, the checkpointer may write a
shared buffer out. This would be considered an IOOP_WRITE IOOp on an
IOOBJECT_RELATION IOObject in the IOCONTEXT_NORMAL IOContext by
BackendType B_CHECKPOINTER.
Each BackendType counts IOOps (evict, extend, fsync, read, reuse, and
write) per IOObject (relation, temp relation) per IOContext (normal,
bulkread, bulkwrite, or vacuum) through a call to pgstat_count_io_op().
Note that this commit introduces the infrastructure to count IO
Operation statistics. A subsequent commit will add calls to
pgstat_count_io_op() in the appropriate locations.
IOObject IOOBJECT_TEMP_RELATION concerns IO Operations on buffers
containing temporary table data, while IOObject IOOBJECT_RELATION
concerns IO Operations on buffers containing permanent relation data.
IOContext IOCONTEXT_NORMAL concerns operations on local and shared
buffers, while IOCONTEXT_BULKREAD, IOCONTEXT_BULKWRITE, and
IOCONTEXT_VACUUM IOContexts concern IO operations on buffers as part of
a BufferAccessStrategy.
Stats on IOOps on all IOObjects in all IOContexts for a given backend
are first counted in a backend's local memory and then flushed to shared
memory and accumulated with those from all other backends, exited and
live.
Some BackendTypes will not flush their pending statistics at regular
intervals and explicitly call pgstat_flush_io_ops() during the course of
normal operations to flush their backend-local IO operation statistics
to shared memory in a timely manner.
Because not all BackendType, IOObject, IOContext, IOOp combinations are
valid, the validity of the stats is checked before flushing pending
stats and before reading in the existing stats file to shared memory.
The aggregated stats in shared memory could be extended in the future
with per-backend stats -- useful for per connection IO statistics and
monitoring.
PGSTAT_FILE_FORMAT_ID should be bumped with this commit.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Reviewed-by: Kyotaro Horiguchi <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 2 +
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 26 ++
src/backend/utils/activity/pgstat_bgwriter.c | 7 +-
.../utils/activity/pgstat_checkpointer.c | 7 +-
src/backend/utils/activity/pgstat_io.c | 386 ++++++++++++++++++
src/backend/utils/activity/pgstat_relation.c | 15 +-
src/backend/utils/activity/pgstat_shmem.c | 4 +
src/backend/utils/activity/pgstat_wal.c | 4 +-
src/backend/utils/adt/pgstatfuncs.c | 11 +-
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 68 +++
src/include/utils/pgstat_internal.h | 30 ++
src/tools/pgindent/typedefs.list | 6 +
15 files changed, 563 insertions(+), 7 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_io.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index e3a783abd0..c47d057a1d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5434,6 +5434,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
the <structname>pg_stat_bgwriter</structname>
view, <literal>archiver</literal> to reset all the counters shown in
the <structname>pg_stat_archiver</structname> view,
+ <literal>io</literal> to reset all the counters shown in the
+ <structname>pg_stat_io</structname> view,
<literal>wal</literal> to reset all the counters shown in the
<structname>pg_stat_wal</structname> view or
<literal>recovery_prefetch</literal> to reset all the counters shown
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index a80eda3cf4..7d7482dde0 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -22,6 +22,7 @@ OBJS = \
pgstat_checkpointer.o \
pgstat_database.o \
pgstat_function.o \
+ pgstat_io.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index a2b872c24b..518ee3f798 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -9,6 +9,7 @@ backend_sources += files(
'pgstat_checkpointer.c',
'pgstat_database.c',
'pgstat_function.c',
+ 'pgstat_io.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 0fa5370bcd..60fc4e761f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -72,6 +72,7 @@
* - pgstat_checkpointer.c
* - pgstat_database.c
* - pgstat_function.c
+ * - pgstat_io.c
* - pgstat_relation.c
* - pgstat_replslot.c
* - pgstat_slru.c
@@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
+ [PGSTAT_KIND_IO] = {
+ .name = "io",
+
+ .fixed_amount = true,
+
+ .reset_all_cb = pgstat_io_reset_all_cb,
+ .snapshot_cb = pgstat_io_snapshot_cb,
+ },
+
[PGSTAT_KIND_SLRU] = {
.name = "slru",
@@ -582,6 +592,7 @@ pgstat_report_stat(bool force)
/* Don't expend a clock check if nothing to do */
if (dlist_is_empty(&pgStatPending) &&
+ !have_iostats &&
!have_slrustats &&
!pgstat_have_pending_wal())
{
@@ -628,6 +639,9 @@ pgstat_report_stat(bool force)
/* flush database / relation / function / ... stats */
partial_flush |= pgstat_flush_pending_entries(nowait);
+ /* flush IO stats */
+ partial_flush |= pgstat_flush_io(nowait);
+
/* flush wal stats */
partial_flush |= pgstat_flush_wal(nowait);
@@ -1322,6 +1336,12 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
+ /*
+ * Write IO stats struct
+ */
+ pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
+ write_chunk_s(fpout, &pgStatLocal.snapshot.io);
+
/*
* Write SLRU stats struct
*/
@@ -1496,6 +1516,12 @@ pgstat_read_statsfile(void)
if (!read_chunk_s(fpin, &shmem->checkpointer.stats))
goto error;
+ /*
+ * Read IO stats struct
+ */
+ if (!read_chunk_s(fpin, &shmem->io.stats))
+ goto error;
+
/*
* Read SLRU stats struct
*/
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 9247f2dda2..92be384b0d 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -24,7 +24,7 @@ PgStat_BgWriterStats PendingBgWriterStats = {0};
/*
- * Report bgwriter statistics
+ * Report bgwriter and IO statistics
*/
void
pgstat_report_bgwriter(void)
@@ -56,6 +56,11 @@ pgstat_report_bgwriter(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 3e9ab45103..26dec112f6 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -24,7 +24,7 @@ PgStat_CheckpointerStats PendingCheckpointerStats = {0};
/*
- * Report checkpointer statistics
+ * Report checkpointer and IO statistics
*/
void
pgstat_report_checkpointer(void)
@@ -62,6 +62,11 @@ pgstat_report_checkpointer(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingCheckpointerStats, 0, sizeof(PendingCheckpointerStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
new file mode 100644
index 0000000000..b606f23eb8
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -0,0 +1,386 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_io.c
+ * Implementation of IO statistics.
+ *
+ * This file contains the implementation of IO statistics. It is kept separate
+ * from pgstat.c to enforce the line between the statistics access / storage
+ * implementation and the details about individual types of statistics.
+ *
+ * Copyright (c) 2021-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_io.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "utils/pgstat_internal.h"
+
+
+static PgStat_BktypeIO PendingIOStats;
+bool have_iostats = false;
+
+/*
+ * Check that stats have not been counted for any combination of IOObject,
+ * IOContext, and IOOp which are not tracked for the passed-in BackendType. The
+ * passed-in PgStat_BktypeIO must contain stats from the BackendType specified
+ * by the second parameter. Caller is responsible for locking the passed-in
+ * PgStat_BktypeIO, if needed.
+ */
+bool
+pgstat_bktype_io_stats_valid(PgStat_BktypeIO *backend_io,
+ BackendType bktype)
+{
+ bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ /*
+ * Don't bother trying to skip to the next loop iteration if
+ * pgstat_tracks_io_object() would return false here. We still
+ * need to validate that each counter is zero anyway.
+ */
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ /* No stats, so nothing to validate */
+ if (backend_io->data[io_object][io_context][io_op] == 0)
+ continue;
+
+ /* There are stats and there shouldn't be */
+ if (!bktype_tracked ||
+ !pgstat_tracks_io_op(bktype, io_object, io_context, io_op))
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
+void
+pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op)
+{
+ Assert(io_object < IOOBJECT_NUM_TYPES);
+ Assert(io_context < IOCONTEXT_NUM_TYPES);
+ Assert(io_op < IOOP_NUM_TYPES);
+ Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
+
+ PendingIOStats.data[io_object][io_context][io_op]++;
+
+ have_iostats = true;
+}
+
+PgStat_IO *
+pgstat_fetch_stat_io(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_IO);
+
+ return &pgStatLocal.snapshot.io;
+}
+
+/*
+ * Flush out locally pending IO statistics
+ *
+ * If no stats have been recorded, this function returns false.
+ *
+ * If nowait is true, this function returns true if the lock could not be
+ * acquired. Otherwise, return false.
+ */
+bool
+pgstat_flush_io(bool nowait)
+{
+ LWLock *bktype_lock;
+ PgStat_BktypeIO *bktype_shstats;
+
+ if (!have_iostats)
+ return false;
+
+ bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
+ bktype_shstats =
+ &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+
+ if (!nowait)
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
+ return true;
+
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOOp io_op = IOOP_FIRST;
+ io_op < IOOP_NUM_TYPES; io_op++)
+ bktype_shstats->data[io_object][io_context][io_op] +=
+ PendingIOStats.data[io_object][io_context][io_op];
+
+ Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
+
+ LWLockRelease(bktype_lock);
+
+ memset(&PendingIOStats, 0, sizeof(PendingIOStats));
+
+ have_iostats = false;
+
+ return false;
+}
+
+const char *
+pgstat_get_io_context_name(IOContext io_context)
+{
+ switch (io_context)
+ {
+ case IOCONTEXT_BULKREAD:
+ return "bulkread";
+ case IOCONTEXT_BULKWRITE:
+ return "bulkwrite";
+ case IOCONTEXT_NORMAL:
+ return "normal";
+ case IOCONTEXT_VACUUM:
+ return "vacuum";
+ }
+
+ elog(ERROR, "unrecognized IOContext value: %d", io_context);
+ pg_unreachable();
+}
+
+const char *
+pgstat_get_io_object_name(IOObject io_object)
+{
+ switch (io_object)
+ {
+ case IOOBJECT_RELATION:
+ return "relation";
+ case IOOBJECT_TEMP_RELATION:
+ return "temp relation";
+ }
+
+ elog(ERROR, "unrecognized IOObject value: %d", io_object);
+ pg_unreachable();
+}
+
+void
+pgstat_io_reset_all_cb(TimestampTz ts)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+
+ memset(bktype_shstats, 0, sizeof(*bktype_shstats));
+ LWLockRelease(bktype_lock);
+ }
+}
+
+void
+pgstat_io_snapshot_cb(void)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_SHARED);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.snapshot.io.stat_reset_timestamp =
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+
+ /* using struct assignment due to better type safety */
+ *bktype_snap = *bktype_shstats;
+ LWLockRelease(bktype_lock);
+ }
+}
+
+/*
+* IO statistics are not collected for all BackendTypes.
+*
+* The following BackendTypes do not participate in the cumulative stats
+* subsystem or do not perform IO on which we currently track:
+* - Syslogger because it is not connected to shared memory
+* - Archiver because most relevant archiving IO is delegated to a
+* specialized command or module
+* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
+*
+* Function returns true if BackendType participates in the cumulative stats
+* subsystem for IO and false if it does not.
+*
+* When adding a new BackendType, also consider adding relevant restrictions to
+* pgstat_tracks_io_object() and pgstat_tracks_io_op().
+*/
+bool
+pgstat_tracks_io_bktype(BackendType bktype)
+{
+ /*
+ * List every type so that new backend types trigger a warning about
+ * needing to adjust this switch.
+ */
+ switch (bktype)
+ {
+ case B_INVALID:
+ case B_ARCHIVER:
+ case B_LOGGER:
+ case B_WAL_RECEIVER:
+ case B_WAL_WRITER:
+ return false;
+
+ case B_AUTOVAC_LAUNCHER:
+ case B_AUTOVAC_WORKER:
+ case B_BACKEND:
+ case B_BG_WORKER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_STANDALONE_BACKEND:
+ case B_STARTUP:
+ case B_WAL_SENDER:
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Some BackendTypes do not perform IO on certain IOObjects or in certain
+ * IOContexts. Some IOObjects are never operated on in some IOContexts. Check
+ * that the given BackendType is expected to do IO in the given IOContext and
+ * on the given IOObject and that the given IOObject is expected to be operated
+ * on in the given IOContext.
+ */
+bool
+pgstat_tracks_io_object(BackendType bktype, IOObject io_object,
+ IOContext io_context)
+{
+ bool no_temp_rel;
+
+ /*
+ * Some BackendTypes should never track IO statistics.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ return false;
+
+ /*
+ * Currently, IO on temporary relations can only occur in the
+ * IOCONTEXT_NORMAL IOContext.
+ */
+ if (io_context != IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * In core Postgres, only regular backends and WAL Sender processes
+ * executing queries will use local buffers and operate on temporary
+ * relations. Parallel workers will not use local buffers (see
+ * InitLocalBuffers()); however, extensions leveraging background workers
+ * have no such limitation, so track IO on IOOBJECT_TEMP_RELATION for
+ * BackendType B_BG_WORKER.
+ */
+ no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
+ bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
+
+ if (no_temp_rel && io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * Some BackendTypes do not currently perform any IO in certain
+ * IOContexts, and, while it may not be inherently incorrect for them to
+ * do so, excluding those rows from the view makes the view easier to use.
+ */
+ if ((bktype == B_CHECKPOINTER || bktype == B_BG_WRITER) &&
+ (io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE ||
+ io_context == IOCONTEXT_VACUUM))
+ return false;
+
+ if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
+ return false;
+
+ if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
+ io_context == IOCONTEXT_BULKWRITE)
+ return false;
+
+ return true;
+}
+
+/*
+ * Some BackendTypes will never do certain IOOps and some IOOps should not
+ * occur in certain IOContexts or on certain IOObjects. Check that the given
+ * IOOp is valid for the given BackendType in the given IOContext and on the
+ * given IOObject. Note that there are currently no cases of an IOOp being
+ * invalid for a particular BackendType only within a certain IOContext and/or
+ * only on a certain IOObject.
+ */
+bool
+pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
+ IOContext io_context, IOOp io_op)
+{
+ bool strategy_io_context;
+
+ /* if (io_context, io_object) will never collect stats, we're done */
+ if (!pgstat_tracks_io_object(bktype, io_object, io_context))
+ return false;
+
+ /*
+ * Some BackendTypes will not do certain IOOps.
+ */
+ if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) &&
+ (io_op == IOOP_READ || io_op == IOOP_EVICT))
+ return false;
+
+ if ((bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER) && io_op == IOOP_EXTEND)
+ return false;
+
+ /*
+ * Some IOOps are not valid in certain IOContexts and some IOOps are only
+ * valid in certain contexts.
+ */
+ if (io_context == IOCONTEXT_BULKREAD && io_op == IOOP_EXTEND)
+ return false;
+
+ strategy_io_context = io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE || io_context == IOCONTEXT_VACUUM;
+
+ /*
+ * IOOP_REUSE is only relevant when a BufferAccessStrategy is in use.
+ */
+ if (!strategy_io_context && io_op == IOOP_REUSE)
+ return false;
+
+ /*
+ * IOOP_FSYNC IOOps done by a backend using a BufferAccessStrategy are
+ * counted in the IOCONTEXT_NORMAL IOContext. See comment in
+ * register_dirty_segment() for more details.
+ */
+ if (strategy_io_context && io_op == IOOP_FSYNC)
+ return false;
+
+ /*
+ * Temporary tables are not logged and thus do not require fsync'ing.
+ */
+ if (io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION && io_op == IOOP_FSYNC)
+ return false;
+
+ return true;
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 2e20b93c20..f793ac1516 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -206,7 +206,7 @@ pgstat_drop_relation(Relation rel)
}
/*
- * Report that the table was just vacuumed.
+ * Report that the table was just vacuumed and flush IO statistics.
*/
void
pgstat_report_vacuum(Oid tableoid, bool shared,
@@ -258,10 +258,18 @@ pgstat_report_vacuum(Oid tableoid, bool shared,
}
pgstat_unlock_entry(entry_ref);
+
+ /*
+ * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
+ * however this will not be called until after an entire autovacuum cycle
+ * is done -- which will likely vacuum many relations -- or until the
+ * VACUUM command has processed all tables and committed.
+ */
+ pgstat_flush_io(false);
}
/*
- * Report that the table was just analyzed.
+ * Report that the table was just analyzed and flush IO statistics.
*
* Caller must provide new live- and dead-tuples estimates, as well as a
* flag indicating whether to reset the mod_since_analyze counter.
@@ -341,6 +349,9 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
+
+ /* see pgstat_report_vacuum() */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index c1506b53d0..09fffd0e82 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -202,6 +202,10 @@ StatsShmemInit(void)
LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&ctl->io.locks[i],
+ LWTRANCHE_PGSTATS_DATA);
}
else
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index e7a82b5fed..e8598b2f4e 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -34,7 +34,7 @@ static WalUsage prevWalUsage;
/*
* Calculate how much WAL usage counters have increased and update
- * shared statistics.
+ * shared WAL and IO statistics.
*
* Must be called by processes that generate WAL, that do not call
* pgstat_report_stat(), like walwriter.
@@ -43,6 +43,8 @@ void
pgstat_report_wal(bool force)
{
pgstat_flush_wal(force);
+
+ pgstat_flush_io(force);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 6737493402..924698e6ae 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1587,7 +1587,12 @@ pg_stat_reset(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
-/* Reset some shared cluster-wide counters */
+/*
+ * Reset some shared cluster-wide counters
+ *
+ * When adding a new reset target, ideally the name should match that in
+ * pgstat_kind_infos, if relevant.
+ */
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
{
@@ -1604,6 +1609,8 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_BGWRITER);
pgstat_reset_of_kind(PGSTAT_KIND_CHECKPOINTER);
}
+ else if (strcmp(target, "io") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_IO);
else if (strcmp(target, "recovery_prefetch") == 0)
XLogPrefetchResetStats();
else if (strcmp(target, "wal") == 0)
@@ -1612,7 +1619,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"io\", \"recovery_prefetch\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 96b3a1e1a0..c309e0233d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -332,6 +332,8 @@ typedef enum BackendType
B_WAL_WRITER,
} BackendType;
+#define BACKEND_NUM_TYPES (B_WAL_WRITER + 1)
+
extern PGDLLIMPORT BackendType MyBackendType;
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5e3326a3b9..9f09caa05f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -48,6 +48,7 @@ typedef enum PgStat_Kind
PGSTAT_KIND_ARCHIVER,
PGSTAT_KIND_BGWRITER,
PGSTAT_KIND_CHECKPOINTER,
+ PGSTAT_KIND_IO,
PGSTAT_KIND_SLRU,
PGSTAT_KIND_WAL,
} PgStat_Kind;
@@ -276,6 +277,55 @@ typedef struct PgStat_CheckpointerStats
PgStat_Counter buf_fsync_backend;
} PgStat_CheckpointerStats;
+
+/*
+ * Types related to counting IO operations
+ */
+typedef enum IOObject
+{
+ IOOBJECT_RELATION,
+ IOOBJECT_TEMP_RELATION,
+} IOObject;
+
+#define IOOBJECT_FIRST IOOBJECT_RELATION
+#define IOOBJECT_NUM_TYPES (IOOBJECT_TEMP_RELATION + 1)
+
+typedef enum IOContext
+{
+ IOCONTEXT_BULKREAD,
+ IOCONTEXT_BULKWRITE,
+ IOCONTEXT_NORMAL,
+ IOCONTEXT_VACUUM,
+} IOContext;
+
+#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
+#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
+
+typedef enum IOOp
+{
+ IOOP_EVICT,
+ IOOP_EXTEND,
+ IOOP_FSYNC,
+ IOOP_READ,
+ IOOP_REUSE,
+ IOOP_WRITE,
+} IOOp;
+
+#define IOOP_FIRST IOOP_EVICT
+#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
+
+typedef struct PgStat_BktypeIO
+{
+ PgStat_Counter data[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
+} PgStat_BktypeIO;
+
+typedef struct PgStat_IO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BktypeIO stats[BACKEND_NUM_TYPES];
+} PgStat_IO;
+
+
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -453,6 +503,24 @@ extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_bktype_io_stats_valid(PgStat_BktypeIO *context_ops,
+ BackendType bktype);
+extern void pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op);
+extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern const char *pgstat_get_io_context_name(IOContext io_context);
+extern const char *pgstat_get_io_object_name(IOObject io_object);
+
+extern bool pgstat_tracks_io_bktype(BackendType bktype);
+extern bool pgstat_tracks_io_object(BackendType bktype,
+ IOObject io_object, IOContext io_context);
+extern bool pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
+ IOContext io_context, IOOp io_op);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 12fd51f1ae..6badb2fde4 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -329,6 +329,17 @@ typedef struct PgStatShared_Checkpointer
PgStat_CheckpointerStats reset_offset;
} PgStatShared_Checkpointer;
+/* Shared-memory ready PgStat_IO */
+typedef struct PgStatShared_IO
+{
+ /*
+ * locks[i] protects stats.stats[i]. locks[0] also protects
+ * stats.stat_reset_timestamp.
+ */
+ LWLock locks[BACKEND_NUM_TYPES];
+ PgStat_IO stats;
+} PgStatShared_IO;
+
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -419,6 +430,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Archiver archiver;
PgStatShared_BgWriter bgwriter;
PgStatShared_Checkpointer checkpointer;
+ PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
} PgStat_ShmemControl;
@@ -442,6 +454,8 @@ typedef struct PgStat_Snapshot
PgStat_CheckpointerStats checkpointer;
+ PgStat_IO io;
+
PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS];
PgStat_WalStats wal;
@@ -549,6 +563,15 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_flush_io(bool nowait);
+extern void pgstat_io_reset_all_cb(TimestampTz ts);
+extern void pgstat_io_snapshot_cb(void);
+
+
/*
* Functions in pgstat_relation.c
*/
@@ -643,6 +666,13 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid)
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
+/*
+ * Variables in pgstat_io.c
+ */
+
+extern PGDLLIMPORT bool have_iostats;
+
+
/*
* Variables in pgstat_slru.c
*/
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 09316039e4..65be0dea1b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1106,7 +1106,10 @@ ID
INFIX
INT128
INTERFACE_INFO
+IOContext
IOFuncSelector
+IOObject
+IOOp
IPCompareMethod
ITEM
IV
@@ -2015,6 +2018,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2024,6 +2028,7 @@ PgStat_ArchiverStats
PgStat_BackendFunctionEntry
PgStat_BackendSubEntry
PgStat_BgWriterStats
+PgStat_BktypeIO
PgStat_CheckpointerStats
PgStat_Counter
PgStat_EntryRef
@@ -2032,6 +2037,7 @@ PgStat_FetchConsistency
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
+PgStat_IO
PgStat_Kind
PgStat_KindInfo
PgStat_LocalState
--
2.34.1
[text/x-patch] v50-0005-pg_stat_io-documentation.patch (13.6K, ../../CAAKRu_b=1cSOq6Ln96GYXaeMfUnVzpjSBXWvi_RNBJbQ0WnXug@mail.gmail.com/4-v50-0005-pg_stat_io-documentation.patch)
download | inline diff:
From a6a90045ae3b68b0fc627e27f740e27d77ea3810 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 17 Jan 2023 16:34:27 -0500
Subject: [PATCH v50 5/5] pg_stat_io documentation
Author: Melanie Plageman <[email protected]>
Author: Samay Sharma <[email protected]>
Reviewed-by: Maciek Sakrejda <[email protected]>
Reviewed-by: Lukas Fittl <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 321 +++++++++++++++++++++++++++++++++--
1 file changed, 307 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c47d057a1d..2f4e6e89bc 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -469,6 +469,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
+ <entry>
+ One row for each combination of backend type, context, and target object
+ containing cluster-wide I/O statistics.
+ See <link linkend="monitoring-pg-stat-io-view">
+ <structname>pg_stat_io</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_replication_slots</structname><indexterm><primary>pg_stat_replication_slots</primary></indexterm></entry>
<entry>One row per replication slot, showing statistics about the
@@ -665,20 +675,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</para>
<para>
- The <structname>pg_statio_</structname> views are primarily useful to
- determine the effectiveness of the buffer cache. When the number
- of actual disk reads is much smaller than the number of buffer
- hits, then the cache is satisfying most read requests without
- invoking a kernel call. However, these statistics do not give the
- entire story: due to the way in which <productname>PostgreSQL</productname>
- handles disk I/O, data that is not in the
- <productname>PostgreSQL</productname> buffer cache might still reside in the
- kernel's I/O cache, and might therefore still be fetched without
- requiring a physical read. Users interested in obtaining more
- detailed information on <productname>PostgreSQL</productname> I/O behavior are
- advised to use the <productname>PostgreSQL</productname> statistics views
- in combination with operating system utilities that allow insight
- into the kernel's handling of I/O.
+ The <structname>pg_stat_io</structname> and
+ <structname>pg_statio_</structname> set of views are useful for determining
+ the effectiveness of the buffer cache. They can be used to calculate a cache
+ hit ratio. Note that while <productname>PostgreSQL</productname>'s I/O
+ statistics capture most instances in which the kernel was invoked in order
+ to perform I/O, they do not differentiate between data which had to be
+ fetched from disk and that which already resided in the kernel page cache.
+ Users are advised to use the <productname>PostgreSQL</productname>
+ statistics views in combination with operating system utilities for a more
+ complete picture of their database's I/O performance.
</para>
</sect2>
@@ -3659,6 +3665,293 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_archived_wal</structfield> have also been successfully
archived.
</para>
+ </sect2>
+
+ <sect2 id="monitoring-pg-stat-io-view">
+ <title><structname>pg_stat_io</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_io</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_io</structname> view will contain one row for each
+ combination of backend type, target I/O object, and I/O context, showing
+ cluster-wide I/O statistics. Combinations which do not make sense are
+ omitted.
+ </para>
+
+ <para>
+ Currently, I/O on relations (e.g. tables, indexes) is tracked. However,
+ relation I/O which bypasses shared buffers (e.g. when moving a table from one
+ tablespace to another) is currently not tracked.
+ </para>
+
+ <table id="pg-stat-io-view" xreflabel="pg_stat_io">
+ <title><structname>pg_stat_io</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>backend_type</structfield> <type>text</type>
+ </para>
+ <para>
+ Type of backend (e.g. background worker, autovacuum worker). See <link
+ linkend="monitoring-pg-stat-activity-view">
+ <structname>pg_stat_activity</structname></link> for more information
+ on <varname>backend_type</varname>s. Some
+ <varname>backend_type</varname>s do not accumulate I/O operation
+ statistics and will not be included in the view.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_object</structfield> <type>text</type>
+ </para>
+ <para>
+ Target object of an I/O operation. Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>relation</literal>: Permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>temp relation</literal>: Temporary relations.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_context</structfield> <type>text</type>
+ </para>
+ <para>
+ The context of an I/O operation. Possible values are:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>normal</literal>: The default or standard
+ <varname>io_context</varname> for a type of I/O operation. For
+ example, by default, relation data is read into and written out from
+ shared buffers. Thus, reads and writes of relation data to and from
+ shared buffers are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>vacuum</literal>: I/O operations performed outside of shared
+ buffers while vacuuming and analyzing permanent relations. Temporary
+ table vacuums use the same local buffer pool as other temporary table
+ IO operations and are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkread</literal>: Certain large read I/O operations
+ done outside of shared buffers, for example, a sequential scan of a
+ large table.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkwrite</literal>: Certain large write I/O operations
+ done outside of shared buffers, such as <command>COPY</command>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reads</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of read operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>writes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of write operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>extends</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of relation extend operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>op_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of bytes per unit of I/O read, written, or extended.
+ </para>
+ <para>
+ Relation data reads, writes, and extends are done in
+ <varname>block_size</varname> units, derived from the build-time
+ parameter <symbol>BLCKSZ</symbol>, which is <literal>8192</literal> by
+ default.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a block has been written out from a shared or local
+ buffer in order to make it available for another use.
+ </para>
+ <para>
+ In <varname>io_context</varname> <literal>normal</literal>, this counts
+ the number of times a block was evicted from a buffer and replaced with
+ another block. In <varname>io_context</varname>s
+ <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
+ <literal>vacuum</literal>, this counts the number of times a block was
+ evicted from shared buffers in order to add the shared buffer to a
+ separate, size-limited ring buffer for use in a bulk I/O operation.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reuses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of times an existing buffer in a size-limited ring buffer
+ outside of shared buffers was reused as part of an I/O operation in the
+ <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
+ <literal>vacuum</literal> <varname>io_context</varname>s.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>fsyncs</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of <literal>fsync</literal> calls. These are only tracked in
+ <varname>io_context</varname> <literal>normal</literal>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset.
+ </para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Some backend types never perform I/O operations on some I/O objects and/or
+ in some I/O contexts. These rows are omitted from the view. For example, the
+ checkpointer does not checkpoint temporary tables, so there will be no rows
+ for <varname>backend_type</varname> <literal>checkpointer</literal> and
+ <varname>io_object</varname> <literal>temp relation</literal>.
+ </para>
+
+ <para>
+ In addition, some I/O operations will never be performed either by certain
+ backend types or on certain I/O objects and/or in certain I/O contexts.
+ These cells will be NULL. For example, temporary tables are not
+ <literal>fsync</literal>ed, so <varname>fsyncs</varname> will be NULL for
+ <varname>io_object</varname> <literal>temp relation</literal>. Also, the
+ background writer does not perform reads, so <varname>reads</varname> will
+ be NULL in rows for <varname>backend_type</varname> <literal>background
+ writer</literal>.
+ </para>
+
+ <para>
+ <structname>pg_stat_io</structname> can be used to inform database tuning.
+ For example:
+ <itemizedlist>
+ <listitem>
+ <para>
+ A high <varname>evictions</varname> count can indicate that shared
+ buffers should be increased.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Client backends rely on the checkpointer to ensure data is persisted to
+ permanent storage. Large numbers of <varname>fsyncs</varname> by
+ <literal>client backend</literal>s could indicate a misconfiguration of
+ shared buffers or of the checkpointer. More information on configuring
+ the checkpointer can be found in <xref linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Normally, client backends should be able to rely on auxiliary processes
+ like the checkpointer and the background writer to write out dirty data
+ as much as possible. Large numbers of writes by client backends could
+ indicate a misconfiguration of shared buffers or of the checkpointer.
+ More information on configuring the checkpointer can be found in <xref
+ linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
</sect2>
--
2.34.1
[text/x-patch] v50-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch (32.0K, ../../CAAKRu_b=1cSOq6Ln96GYXaeMfUnVzpjSBXWvi_RNBJbQ0WnXug@mail.gmail.com/5-v50-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch)
download | inline diff:
From 471d98971d39d33a9e75bc1463f8f7d6c2973dfe Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 17 Jan 2023 16:28:27 -0500
Subject: [PATCH v50 4/5] Add system view tracking IO ops per backend type
Add pg_stat_io, a system view which tracks the number of IOOps
(evictions, reuses, reads, writes, extensions, and fsyncs) done on each
IOObject (relation, temp relation) in each IOContext ("normal" and those
using a BufferAccessStrategy) by each type of backend (e.g. client
backend, checkpointer).
Some BackendTypes do not accumulate IO operations statistics and will
not be included in the view.
Some IOObjects are never operated on in some IOContexts or by some
BackendTypes. These rows are omitted from the view. For example,
checkpointer will never operate on IOOBJECT_TEMP_RELATION data, so those
rows are omitted.
Some IOContexts are not used by some BackendTypes and will not be in the
view. For example, checkpointer does not use a BufferAccessStrategy
(currently), so there will be no rows for BufferAccessStrategy
IOContexts for checkpointer.
Some IOOps are invalid in combination with certain IOContexts and
certain IOObjects. Those cells will be NULL in the view to distinguish
between 0 observed IOOps of that type and an invalid combination. For
example, temporary tables are not fsynced so cells for all BackendTypes
for IOOBJECT_TEMP_RELATION and IOOP_FSYNC will be NULL.
Some BackendTypes never perform certain IOOps. Those cells will also be
NULL in the view. For example, bgwriter should not perform reads.
View stats are populated with statistics incremented when a backend
performs an IO Operation and maintained by the cumulative statistics
subsystem.
Each row of the view shows stats for a particular BackendType, IOObject,
IOContext combination (e.g. a client backend's operations on permanent
relations in shared buffers) and each column in the view is the total
number of IO Operations done (e.g. writes). So a cell in the view would
be, for example, the number of blocks of relation data written from
shared buffers by client backends since the last stats reset.
In anticipation of tracking WAL IO and non-block-oriented IO (such as
temporary file IO), the "op_bytes" column specifies the unit of the
"reads", "writes", and "extends" columns for a given row.
Note that some of the cells in the view are redundant with fields in
pg_stat_bgwriter (e.g. buffers_backend), however these have been kept in
pg_stat_bgwriter for backwards compatibility. Deriving the redundant
pg_stat_bgwriter stats from the IO operations stats structures was also
problematic due to the separate reset targets for 'bgwriter' and 'io'.
Suggested by Andres Freund
Catalog version should be bumped.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
contrib/amcheck/expected/check_heap.out | 34 ++++
contrib/amcheck/sql/check_heap.sql | 27 +++
src/backend/catalog/system_views.sql | 15 ++
src/backend/utils/adt/pgstatfuncs.c | 141 +++++++++++++++
src/include/catalog/pg_proc.dat | 9 +
src/test/regress/expected/rules.out | 12 ++
src/test/regress/expected/stats.out | 227 ++++++++++++++++++++++++
src/test/regress/sql/stats.sql | 141 +++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 607 insertions(+)
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index c010361025..e4785141a6 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -66,6 +66,22 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -88,6 +104,23 @@ SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock :=
-------+--------+--------+-----
(0 rows)
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+ ?column?
+----------
+ t
+(1 row)
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
SET ROLE regress_heaptest_role;
@@ -195,6 +228,7 @@ ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 298de6886a..6794ca4eb0 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -20,11 +20,29 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
+
-- Add some data so subsequent tests are not entirely trivial
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
+
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -32,6 +50,14 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');
SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
@@ -110,6 +136,7 @@ SELECT * FROM verify_heapam('test_foreign_table',
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..34ca0e739f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1117,6 +1117,21 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+CREATE VIEW pg_stat_io AS
+SELECT
+ b.backend_type,
+ b.io_object,
+ b.io_context,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+FROM pg_stat_get_io() b;
+
CREATE VIEW pg_stat_wal AS
SELECT
w.wal_records,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 924698e6ae..9d707c3521 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1245,6 +1245,147 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+* When adding a new column to the pg_stat_io view, add a new enum value
+* here above IO_NUM_COLUMNS.
+*/
+typedef enum io_stat_col
+{
+ IO_COL_BACKEND_TYPE,
+ IO_COL_IO_OBJECT,
+ IO_COL_IO_CONTEXT,
+ IO_COL_READS,
+ IO_COL_WRITES,
+ IO_COL_EXTENDS,
+ IO_COL_CONVERSION,
+ IO_COL_EVICTIONS,
+ IO_COL_REUSES,
+ IO_COL_FSYNCS,
+ IO_COL_RESET_TIME,
+ IO_NUM_COLUMNS,
+} io_stat_col;
+
+/*
+ * When adding a new IOOp, add a new io_stat_col and add a case to this
+ * function returning the corresponding io_stat_col.
+ */
+static io_stat_col
+pgstat_get_io_op_index(IOOp io_op)
+{
+ switch (io_op)
+ {
+ case IOOP_EVICT:
+ return IO_COL_EVICTIONS;
+ case IOOP_READ:
+ return IO_COL_READS;
+ case IOOP_REUSE:
+ return IO_COL_REUSES;
+ case IOOP_WRITE:
+ return IO_COL_WRITES;
+ case IOOP_EXTEND:
+ return IO_COL_EXTENDS;
+ case IOOP_FSYNC:
+ return IO_COL_FSYNCS;
+ }
+
+ elog(ERROR, "unrecognized IOOp value: %d", io_op);
+ pg_unreachable();
+}
+
+Datum
+pg_stat_get_io(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo;
+ PgStat_IO *backends_io_stats;
+ Datum reset_time;
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ backends_io_stats = pgstat_fetch_stat_io();
+
+ reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
+
+ for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ {
+ Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
+ PgStat_BktypeIO *bktype_stats = &backends_io_stats->stats[bktype];
+
+ /*
+ * In Assert builds, we can afford an extra loop through all of the
+ * counters checking that only expected stats are non-zero, since it
+ * keeps the non-Assert code cleaner.
+ */
+ Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype));
+
+ /*
+ * For those BackendTypes without IO Operation stats, skip
+ * representing them in the view altogether.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ continue;
+
+ for (IOObject io_obj = IOOBJECT_FIRST;
+ io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ {
+ const char *obj_name = pgstat_get_io_object_name(io_obj);
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ const char *context_name = pgstat_get_io_context_name(io_context);
+
+ Datum values[IO_NUM_COLUMNS] = {0};
+ bool nulls[IO_NUM_COLUMNS] = {0};
+
+ /*
+ * Some combinations of BackendType, IOObject, and IOContext
+ * are not valid for any type of IOOp. In such cases, omit the
+ * entire row from the view.
+ */
+ if (!pgstat_tracks_io_object(bktype, io_obj, io_context))
+ continue;
+
+ values[IO_COL_BACKEND_TYPE] = bktype_desc;
+ values[IO_COL_IO_CONTEXT] = CStringGetTextDatum(context_name);
+ values[IO_COL_IO_OBJECT] = CStringGetTextDatum(obj_name);
+ values[IO_COL_RESET_TIME] = TimestampTzGetDatum(reset_time);
+
+ /*
+ * Hard-code this to the value of BLCKSZ for now. Future
+ * values could include XLOG_BLCKSZ, once WAL IO is tracked,
+ * and constant multipliers, once non-block-oriented IO (e.g.
+ * temporary file IO) is tracked.
+ */
+ values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
+
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ int col_idx = pgstat_get_io_op_index(io_op);
+
+ /*
+ * Some combinations of BackendType and IOOp, of IOContext
+ * and IOOp, and of IOObject and IOOp are not tracked. Set
+ * these cells in the view NULL.
+ */
+ nulls[col_idx] = !pgstat_tracks_io_op(bktype, io_obj, io_context, io_op);
+
+ if (nulls[col_idx])
+ continue;
+
+ values[col_idx] =
+ Int64GetDatum(bktype_stats->data[io_obj][io_context][io_op]);
+ }
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ }
+ }
+
+ return (Datum) 0;
+}
+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 86eb8e8c58..2e804c5bd4 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5690,6 +5690,15 @@
proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
+{ oid => '8459', descr => 'statistics: per backend type IO statistics',
+ proname => 'pg_stat_get_io', provolatile => 'v',
+ prorows => '30', proretset => 't',
+ proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,text,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{backend_type,io_object,io_context,reads,writes,extends,op_bytes,evictions,reuses,fsyncs,stats_reset}',
+ prosrc => 'pg_stat_get_io' },
+
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e7a2f5856a..174b725fff 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1876,6 +1876,18 @@ pg_stat_gssapi| SELECT pid,
gss_enc AS encrypted
FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
WHERE (client_port IS NOT NULL);
+pg_stat_io| SELECT backend_type,
+ io_object,
+ io_context,
+ reads,
+ writes,
+ extends,
+ op_bytes,
+ evictions,
+ reuses,
+ fsyncs,
+ stats_reset
+ FROM pg_stat_get_io() b(backend_type, io_object, io_context, reads, writes, extends, op_bytes, evictions, reuses, fsyncs, stats_reset);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 1d84407a03..3ad38da0dd 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1126,4 +1126,231 @@ SELECT pg_stat_get_subscription_stats(NULL);
(1 row)
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+ ?column?
+----------
+ t
+(1 row)
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_tblspace;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+ count
+-------
+ 100
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+ count
+-------
+ 5000
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+ ?column? | ?column? | ?column? | ?column?
+----------+----------+----------+----------
+ t | t | t | t
+(1 row)
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_tblspace;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+RESET temp_buffers;
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
+
+RESET wal_skip_threshold;
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+ pg_stat_have_stats
+--------------------
+ t
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+ ?column?
+----------
+ t
+(1 row)
+
-- End of Stats Test
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index b4d6753c71..5badd09a1c 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -536,4 +536,145 @@ SELECT pg_stat_get_replication_slot(NULL);
SELECT pg_stat_get_subscription_stats(NULL);
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_tblspace;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+SELECT pg_stat_force_next_flush();
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_tblspace;
+SELECT pg_stat_force_next_flush();
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+RESET temp_buffers;
+
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+RESET wal_skip_threshold;
+
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+
-- End of Stats Test
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 65be0dea1b..970a0cfd1d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3376,6 +3376,7 @@ intset_internal_node
intset_leaf_node
intset_node
intvKEY
+io_stat_col
itemIdCompact
itemIdCompactData
iterator
--
2.34.1
[text/x-patch] v50-0003-pgstat-Count-IO-for-relations.patch (22.2K, ../../CAAKRu_b=1cSOq6Ln96GYXaeMfUnVzpjSBXWvi_RNBJbQ0WnXug@mail.gmail.com/6-v50-0003-pgstat-Count-IO-for-relations.patch)
download | inline diff:
From 2c138faa33ff93756b6bda70e68798c6ab4afbe6 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 17 Jan 2023 16:25:31 -0500
Subject: [PATCH v50 3/5] pgstat: Count IO for relations
Count IOOps done on IOObjects in IOContexts by various BackendTypes
using the IO stats infrastructure introduced by a previous commit.
The primary concern of these statistics is IO operations on data blocks
during the course of normal database operations. IO operations done by,
for example, the archiver or syslogger are not counted in these
statistics. WAL IO, temporary file IO, and IO done directly though smgr*
functions (such as when building an index) are not yet counted but would
be useful future additions.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
src/backend/storage/buffer/bufmgr.c | 111 ++++++++++++++++++++++----
src/backend/storage/buffer/freelist.c | 58 ++++++++++----
src/backend/storage/buffer/localbuf.c | 13 ++-
src/backend/storage/smgr/md.c | 24 ++++++
src/include/storage/buf_internals.h | 8 +-
src/include/storage/bufmgr.h | 7 +-
6 files changed, 185 insertions(+), 36 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8075828e8a..ff12bc2ba6 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -481,8 +481,9 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr);
-static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
+ bool *foundPtr, IOContext *io_context);
+static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
+ IOObject io_object, IOContext io_context);
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -823,6 +824,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BufferDesc *bufHdr;
Block bufBlock;
bool found;
+ IOContext io_context;
+ IOObject io_object;
bool isExtend;
bool isLocalBuf = SmgrIsTemp(smgr);
@@ -855,7 +858,14 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isLocalBuf)
{
- bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found);
+ /*
+ * LocalBufferAlloc() will set the io_context to IOCONTEXT_NORMAL. We
+ * do not use a BufferAccessStrategy for I/O of temporary tables.
+ * However, in some cases, the "strategy" may not be NULL, so we can't
+ * rely on IOContextForStrategy() to set the right IOContext for us.
+ * This may happen in cases like CREATE TEMPORARY TABLE AS...
+ */
+ bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found, &io_context);
if (found)
pgBufferUsage.local_blks_hit++;
else if (isExtend)
@@ -871,7 +881,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, &found, &io_context);
if (found)
pgBufferUsage.shared_blks_hit++;
else if (isExtend)
@@ -986,7 +996,16 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
Assert(!(pg_atomic_read_u32(&bufHdr->state) & BM_VALID)); /* spinlock not needed */
- bufBlock = isLocalBuf ? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr);
+ if (isLocalBuf)
+ {
+ bufBlock = LocalBufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ bufBlock = BufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_RELATION;
+ }
if (isExtend)
{
@@ -995,6 +1014,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
/* don't set checksum for all-zero page */
smgrextend(smgr, forkNum, blockNum, (char *) bufBlock, false);
+ pgstat_count_io_op(io_object, io_context, IOOP_EXTEND);
+
/*
* NB: we're *not* doing a ScheduleBufferTagForWriteback here;
* although we're essentially performing a write. At least on linux
@@ -1020,6 +1041,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgrread(smgr, forkNum, blockNum, (char *) bufBlock);
+ pgstat_count_io_op(io_object, io_context, IOOP_READ);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -1113,14 +1136,19 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* *foundPtr is actually redundant with the buffer's BM_VALID flag, but
* we keep it for simplicity in ReadBuffer.
*
+ * io_context is passed as an output parameter to avoid calling
+ * IOContextForStrategy() when there is a shared buffers hit and no IO
+ * statistics need be captured.
+ *
* No locks are held either at entry or exit.
*/
static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
+ bool from_ring;
BufferTag newTag; /* identity of requested block */
uint32 newHash; /* hash value for newTag */
LWLock *newPartitionLock; /* buffer partition lock for it */
@@ -1172,8 +1200,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* If we get here, previous attempts to read the buffer must
- * have failed ... but we shall bravely try again.
+ * have failed ... but we shall bravely try again. Set
+ * io_context since we will in fact need to count an IO
+ * Operation.
*/
+ *io_context = IOContextForStrategy(strategy);
*foundPtr = false;
}
}
@@ -1187,6 +1218,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ *io_context = IOContextForStrategy(strategy);
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1200,7 +1233,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* Select a victim buffer. The buffer is returned with its header
* spinlock still held!
*/
- buf = StrategyGetBuffer(strategy, &buf_state);
+ buf = StrategyGetBuffer(strategy, &buf_state, &from_ring);
Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 0);
@@ -1254,7 +1287,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
if (XLogNeedsFlush(lsn) &&
- StrategyRejectBuffer(strategy, buf))
+ StrategyRejectBuffer(strategy, buf, from_ring))
{
/* Drop lock/pin and loop around for another buffer */
LWLockRelease(BufferDescriptorGetContentLock(buf));
@@ -1269,7 +1302,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgr->smgr_rlocator.locator.dbOid,
smgr->smgr_rlocator.locator.relNumber);
- FlushBuffer(buf, NULL);
+ FlushBuffer(buf, NULL, IOOBJECT_RELATION, *io_context);
LWLockRelease(BufferDescriptorGetContentLock(buf));
ScheduleBufferTagForWriteback(&BackendWritebackContext,
@@ -1450,6 +1483,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
LWLockRelease(newPartitionLock);
+ if (oldFlags & BM_VALID)
+ {
+ /*
+ * When a BufferAccessStrategy is in use, blocks evicted from shared
+ * buffers are counted as IOOP_EVICT in the corresponding context
+ * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
+ * strategy in two cases: 1) while initially claiming buffers for the
+ * strategy ring 2) to replace an existing strategy ring buffer
+ * because it is pinned or in use and cannot be reused.
+ *
+ * Blocks evicted from buffers already in the strategy ring are
+ * counted as IOOP_REUSE in the corresponding strategy context.
+ *
+ * At this point, we can accurately count evictions and reuses,
+ * because we have successfully claimed the valid buffer. Previously,
+ * we may have been forced to release the buffer due to concurrent
+ * pinners or erroring out.
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, *io_context,
+ from_ring ? IOOP_REUSE : IOOP_EVICT);
+ }
+
/*
* Buffer contents are currently invalid. Try to obtain the right to
* start I/O. If StartBufferIO returns false, then someone else managed
@@ -2570,7 +2625,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
@@ -2820,7 +2875,8 @@ BufferGetTag(Buffer buffer, RelFileLocator *rlocator, ForkNumber *forknum,
* as the second parameter. If not, pass NULL.
*/
static void
-FlushBuffer(BufferDesc *buf, SMgrRelation reln)
+FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
+ IOContext io_context)
{
XLogRecPtr recptr;
ErrorContextCallback errcallback;
@@ -2912,6 +2968,26 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
bufToWrite,
false);
+ /*
+ * When a strategy is in use, only flushes of dirty buffers already in the
+ * strategy ring are counted as strategy writes (IOCONTEXT
+ * [BULKREAD|BULKWRITE|VACUUM] IOOP_WRITE) for the purpose of IO
+ * statistics tracking.
+ *
+ * If a shared buffer initially added to the ring must be flushed before
+ * being used, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE.
+ *
+ * If a shared buffer which was added to the ring later because the
+ * current strategy buffer is pinned or in use or because all strategy
+ * buffers were dirty and rejected (for BAS_BULKREAD operations only)
+ * requires flushing, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE
+ * (from_ring will be false).
+ *
+ * When a strategy is not in use, the write can only be a "regular" write
+ * of a dirty shared buffer (IOCONTEXT_NORMAL IOOP_WRITE).
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, io_context, IOOP_WRITE);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -3554,6 +3630,8 @@ FlushRelationBuffers(Relation rel)
buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_WRITE);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -3586,7 +3664,8 @@ FlushRelationBuffers(Relation rel)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, RelationGetSmgr(rel));
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOCONTEXT_NORMAL, IOOBJECT_RELATION);
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3684,7 +3763,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, srelent->srel);
+ FlushBuffer(bufHdr, srelent->srel, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3894,7 +3973,7 @@ FlushDatabaseBuffers(Oid dbid)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3921,7 +4000,7 @@ FlushOneBuffer(Buffer buffer)
Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr)));
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7dec35801c..c690d5f15f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,7 @@
*/
#include "postgres.h"
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -81,12 +82,6 @@ typedef struct BufferAccessStrategyData
*/
int current;
- /*
- * True if the buffer just returned by StrategyGetBuffer had been in the
- * ring already.
- */
- bool current_was_in_ring;
-
/*
* Array of buffer numbers. InvalidBuffer (that is, zero) indicates we
* have not yet selected a buffer for this ring slot. For allocation
@@ -198,13 +193,15 @@ have_free_buffer(void)
* return the buffer with the buffer header spinlock still held.
*/
BufferDesc *
-StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
+StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring)
{
BufferDesc *buf;
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ *from_ring = false;
+
/*
* If given a strategy object, see whether it can select a buffer. We
* assume strategy objects don't need buffer_strategy_lock.
@@ -213,7 +210,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
{
buf = GetBufferFromRing(strategy, buf_state);
if (buf != NULL)
+ {
+ *from_ring = true;
return buf;
+ }
}
/*
@@ -602,7 +602,7 @@ FreeAccessStrategy(BufferAccessStrategy strategy)
/*
* GetBufferFromRing -- returns a buffer from the ring, or NULL if the
- * ring is empty.
+ * ring is empty / not usable.
*
* The bufhdr spin lock is held on the returned buffer.
*/
@@ -625,10 +625,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
*/
bufnum = strategy->buffers[strategy->current];
if (bufnum == InvalidBuffer)
- {
- strategy->current_was_in_ring = false;
return NULL;
- }
/*
* If the buffer is pinned we cannot use it under any circumstances.
@@ -644,7 +641,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) <= 1)
{
- strategy->current_was_in_ring = true;
*buf_state = local_buf_state;
return buf;
}
@@ -654,7 +650,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
* Tell caller to allocate a new buffer with the normal allocation
* strategy. He'll then replace this ring element via AddBufferToRing.
*/
- strategy->current_was_in_ring = false;
return NULL;
}
@@ -670,6 +665,39 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
strategy->buffers[strategy->current] = BufferDescriptorGetBuffer(buf);
}
+/*
+ * Utility function returning the IOContext of a given BufferAccessStrategy's
+ * strategy ring.
+ */
+IOContext
+IOContextForStrategy(BufferAccessStrategy strategy)
+{
+ if (!strategy)
+ return IOCONTEXT_NORMAL;
+
+ switch (strategy->btype)
+ {
+ case BAS_NORMAL:
+
+ /*
+ * Currently, GetAccessStrategy() returns NULL for
+ * BufferAccessStrategyType BAS_NORMAL, so this case is
+ * unreachable.
+ */
+ pg_unreachable();
+ return IOCONTEXT_NORMAL;
+ case BAS_BULKREAD:
+ return IOCONTEXT_BULKREAD;
+ case BAS_BULKWRITE:
+ return IOCONTEXT_BULKWRITE;
+ case BAS_VACUUM:
+ return IOCONTEXT_VACUUM;
+ }
+
+ elog(ERROR, "unrecognized BufferAccessStrategyType: %d", strategy->btype);
+ pg_unreachable();
+}
+
/*
* StrategyRejectBuffer -- consider rejecting a dirty buffer
*
@@ -682,14 +710,14 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
* if this buffer should be written and re-used.
*/
bool
-StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
+StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring)
{
/* We only do this in bulkread mode */
if (strategy->btype != BAS_BULKREAD)
return false;
/* Don't muck with behavior of normal buffer-replacement strategy */
- if (!strategy->current_was_in_ring ||
+ if (!from_ring ||
strategy->buffers[strategy->current] != BufferDescriptorGetBuffer(buf))
return false;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8372acc383..8e286db5df 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "catalog/catalog.h"
#include "executor/instrument.h"
+#include "pgstat.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "utils/guc_hooks.h"
@@ -107,7 +108,7 @@ PrefetchLocalBuffer(SMgrRelation smgr, ForkNumber forkNum,
*/
BufferDesc *
LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
BufferTag newTag; /* identity of requested block */
LocalBufferLookupEnt *hresult;
@@ -127,6 +128,14 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
+ /*
+ * IO Operations on local buffers are only done in IOCONTEXT_NORMAL. Set
+ * io_context here (instead of after a buffer hit would have returned) for
+ * convenience since we don't have to worry about the overhead of calling
+ * IOContextForStrategy().
+ */
+ *io_context = IOCONTEXT_NORMAL;
+
if (hresult)
{
b = hresult->id;
@@ -230,6 +239,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
buf_state &= ~BM_DIRTY;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_WRITE);
pgBufferUsage.local_blks_written++;
}
@@ -256,6 +266,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
ClearBufferTag(&bufHdr->tag);
buf_state &= ~(BM_VALID | BM_TAG_VALID);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_EVICT);
}
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 60c9905eff..8da813600c 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -983,6 +983,15 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
{
MdfdVec *v = &reln->md_seg_fds[forknum][segno - 1];
+ /*
+ * fsyncs done through mdimmedsync() should be tracked in a separate
+ * IOContext than those done through mdsyncfiletag() to differentiate
+ * between unavoidable client backend fsyncs (e.g. those done during
+ * index build) and those which ideally would have been done by the
+ * checkpointer. Since other IO operations bypassing the buffer
+ * manager could also be tracked in such an IOContext, wait until
+ * these are also tracked to track immediate fsyncs.
+ */
if (FileSync(v->mdfd_vfd, WAIT_EVENT_DATA_FILE_IMMEDIATE_SYNC) < 0)
ereport(data_sync_elevel(ERROR),
(errcode_for_file_access(),
@@ -1021,6 +1030,19 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
{
+ /*
+ * We have no way of knowing if the current IOContext is
+ * IOCONTEXT_NORMAL or IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] at this
+ * point, so count the fsync as being in the IOCONTEXT_NORMAL
+ * IOContext. This is probably okay, because the number of backend
+ * fsyncs doesn't say anything about the efficacy of the
+ * BufferAccessStrategy. And counting both fsyncs done in
+ * IOCONTEXT_NORMAL and IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] under
+ * IOCONTEXT_NORMAL is likely clearer when investigating the number of
+ * backend fsyncs.
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL, IOOP_FSYNC);
+
ereport(DEBUG1,
(errmsg_internal("could not forward fsync request because request queue is full")));
@@ -1410,6 +1432,8 @@ mdsyncfiletag(const FileTag *ftag, char *path)
if (need_to_close)
FileClose(file);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL, IOOP_FSYNC);
+
errno = save_errno;
return result;
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index ed8aa2519c..0b44814740 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -15,6 +15,7 @@
#ifndef BUFMGR_INTERNALS_H
#define BUFMGR_INTERNALS_H
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
@@ -391,11 +392,12 @@ extern void IssuePendingWritebacks(WritebackContext *context);
extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *tag);
/* freelist.c */
+extern IOContext IOContextForStrategy(BufferAccessStrategy bas);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
- uint32 *buf_state);
+ uint32 *buf_state, bool *from_ring);
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
- BufferDesc *buf);
+ BufferDesc *buf, bool from_ring);
extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
extern void StrategyNotifyBgWriter(int bgwprocno);
@@ -417,7 +419,7 @@ extern PrefetchBufferResult PrefetchLocalBuffer(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum);
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
- BlockNumber blockNum, bool *foundPtr);
+ BlockNumber blockNum, bool *foundPtr, IOContext *io_context);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 33eadbc129..b8a18b8081 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -23,7 +23,12 @@
typedef void *Block;
-/* Possible arguments for GetAccessStrategy() */
+/*
+ * Possible arguments for GetAccessStrategy().
+ *
+ * If adding a new BufferAccessStrategyType, also add a new IOContext so
+ * IO statistics using this strategy are tracked.
+ */
typedef enum BufferAccessStrategyType
{
BAS_NORMAL, /* Normal random access */
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-20 02:15 ` Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Melanie Plageman @ 2023-01-20 02:15 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Andres Freund <[email protected]>; Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>
On Thu, Jan 19, 2023 at 4:28 PM Melanie Plageman
<[email protected]> wrote:
>
> On Thu, Jan 19, 2023 at 6:18 AM vignesh C <[email protected]> wrote:
> > The patch does not apply on top of HEAD as in [1], please post a rebased patch:
> > === Applying patches on top of PostgreSQL commit ID
> > 4f74f5641d53559ec44e74d5bf552e167fdd5d20 ===
> > === applying patch
> > ./v49-0003-Add-system-view-tracking-IO-ops-per-backend-type.patch
> > ....
> > patching file src/test/regress/expected/rules.out
> > Hunk #1 FAILED at 1876.
> > 1 out of 1 hunk FAILED -- saving rejects to file
> > src/test/regress/expected/rules.out.rej
> >
> > [1] - http://cfbot.cputube.org/patch_41_3272.log
>
> Yes, it conflicted with 47bb9db75996232. rebased v50 is attached.
Oh dear-- an extra FlushBuffer() snuck in there somehow.
Removed it in attached v51.
Also, I fixed an issue in my tablespace.sql updates
- Melanie
Attachments:
[text/x-patch] v51-0005-pg_stat_io-documentation.patch (13.6K, ../../CAAKRu_af0YcV9CXG9M5R06tYRUzk13+H7qOsU6etmS4zE7RAUQ@mail.gmail.com/2-v51-0005-pg_stat_io-documentation.patch)
download | inline diff:
From f477dfb566a47350bc78dcfb925db429e79fd657 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 17 Jan 2023 16:34:27 -0500
Subject: [PATCH v51 5/5] pg_stat_io documentation
Author: Melanie Plageman <[email protected]>
Author: Samay Sharma <[email protected]>
Reviewed-by: Maciek Sakrejda <[email protected]>
Reviewed-by: Lukas Fittl <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 321 +++++++++++++++++++++++++++++++++--
1 file changed, 307 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c47d057a1d..2f4e6e89bc 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -469,6 +469,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_io</structname><indexterm><primary>pg_stat_io</primary></indexterm></entry>
+ <entry>
+ One row for each combination of backend type, context, and target object
+ containing cluster-wide I/O statistics.
+ See <link linkend="monitoring-pg-stat-io-view">
+ <structname>pg_stat_io</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_replication_slots</structname><indexterm><primary>pg_stat_replication_slots</primary></indexterm></entry>
<entry>One row per replication slot, showing statistics about the
@@ -665,20 +675,16 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</para>
<para>
- The <structname>pg_statio_</structname> views are primarily useful to
- determine the effectiveness of the buffer cache. When the number
- of actual disk reads is much smaller than the number of buffer
- hits, then the cache is satisfying most read requests without
- invoking a kernel call. However, these statistics do not give the
- entire story: due to the way in which <productname>PostgreSQL</productname>
- handles disk I/O, data that is not in the
- <productname>PostgreSQL</productname> buffer cache might still reside in the
- kernel's I/O cache, and might therefore still be fetched without
- requiring a physical read. Users interested in obtaining more
- detailed information on <productname>PostgreSQL</productname> I/O behavior are
- advised to use the <productname>PostgreSQL</productname> statistics views
- in combination with operating system utilities that allow insight
- into the kernel's handling of I/O.
+ The <structname>pg_stat_io</structname> and
+ <structname>pg_statio_</structname> set of views are useful for determining
+ the effectiveness of the buffer cache. They can be used to calculate a cache
+ hit ratio. Note that while <productname>PostgreSQL</productname>'s I/O
+ statistics capture most instances in which the kernel was invoked in order
+ to perform I/O, they do not differentiate between data which had to be
+ fetched from disk and that which already resided in the kernel page cache.
+ Users are advised to use the <productname>PostgreSQL</productname>
+ statistics views in combination with operating system utilities for a more
+ complete picture of their database's I/O performance.
</para>
</sect2>
@@ -3659,6 +3665,293 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<structfield>last_archived_wal</structfield> have also been successfully
archived.
</para>
+ </sect2>
+
+ <sect2 id="monitoring-pg-stat-io-view">
+ <title><structname>pg_stat_io</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_io</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_io</structname> view will contain one row for each
+ combination of backend type, target I/O object, and I/O context, showing
+ cluster-wide I/O statistics. Combinations which do not make sense are
+ omitted.
+ </para>
+
+ <para>
+ Currently, I/O on relations (e.g. tables, indexes) is tracked. However,
+ relation I/O which bypasses shared buffers (e.g. when moving a table from one
+ tablespace to another) is currently not tracked.
+ </para>
+
+ <table id="pg-stat-io-view" xreflabel="pg_stat_io">
+ <title><structname>pg_stat_io</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>backend_type</structfield> <type>text</type>
+ </para>
+ <para>
+ Type of backend (e.g. background worker, autovacuum worker). See <link
+ linkend="monitoring-pg-stat-activity-view">
+ <structname>pg_stat_activity</structname></link> for more information
+ on <varname>backend_type</varname>s. Some
+ <varname>backend_type</varname>s do not accumulate I/O operation
+ statistics and will not be included in the view.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_object</structfield> <type>text</type>
+ </para>
+ <para>
+ Target object of an I/O operation. Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>relation</literal>: Permanent relations.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>temp relation</literal>: Temporary relations.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>io_context</structfield> <type>text</type>
+ </para>
+ <para>
+ The context of an I/O operation. Possible values are:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>normal</literal>: The default or standard
+ <varname>io_context</varname> for a type of I/O operation. For
+ example, by default, relation data is read into and written out from
+ shared buffers. Thus, reads and writes of relation data to and from
+ shared buffers are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>vacuum</literal>: I/O operations performed outside of shared
+ buffers while vacuuming and analyzing permanent relations. Temporary
+ table vacuums use the same local buffer pool as other temporary table
+ IO operations and are tracked in <varname>io_context</varname>
+ <literal>normal</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkread</literal>: Certain large read I/O operations
+ done outside of shared buffers, for example, a sequential scan of a
+ large table.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>bulkwrite</literal>: Certain large write I/O operations
+ done outside of shared buffers, such as <command>COPY</command>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reads</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of read operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>writes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of write operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>extends</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of relation extend operations, each of the size specified in
+ <varname>op_bytes</varname>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>op_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of bytes per unit of I/O read, written, or extended.
+ </para>
+ <para>
+ Relation data reads, writes, and extends are done in
+ <varname>block_size</varname> units, derived from the build-time
+ parameter <symbol>BLCKSZ</symbol>, which is <literal>8192</literal> by
+ default.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a block has been written out from a shared or local
+ buffer in order to make it available for another use.
+ </para>
+ <para>
+ In <varname>io_context</varname> <literal>normal</literal>, this counts
+ the number of times a block was evicted from a buffer and replaced with
+ another block. In <varname>io_context</varname>s
+ <literal>bulkwrite</literal>, <literal>bulkread</literal>, and
+ <literal>vacuum</literal>, this counts the number of times a block was
+ evicted from shared buffers in order to add the shared buffer to a
+ separate, size-limited ring buffer for use in a bulk I/O operation.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>reuses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of times an existing buffer in a size-limited ring buffer
+ outside of shared buffers was reused as part of an I/O operation in the
+ <literal>bulkread</literal>, <literal>bulkwrite</literal>, or
+ <literal>vacuum</literal> <varname>io_context</varname>s.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>fsyncs</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of <literal>fsync</literal> calls. These are only tracked in
+ <varname>io_context</varname> <literal>normal</literal>.
+ </para>
+ </entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry">
+ <para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset.
+ </para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Some backend types never perform I/O operations on some I/O objects and/or
+ in some I/O contexts. These rows are omitted from the view. For example, the
+ checkpointer does not checkpoint temporary tables, so there will be no rows
+ for <varname>backend_type</varname> <literal>checkpointer</literal> and
+ <varname>io_object</varname> <literal>temp relation</literal>.
+ </para>
+
+ <para>
+ In addition, some I/O operations will never be performed either by certain
+ backend types or on certain I/O objects and/or in certain I/O contexts.
+ These cells will be NULL. For example, temporary tables are not
+ <literal>fsync</literal>ed, so <varname>fsyncs</varname> will be NULL for
+ <varname>io_object</varname> <literal>temp relation</literal>. Also, the
+ background writer does not perform reads, so <varname>reads</varname> will
+ be NULL in rows for <varname>backend_type</varname> <literal>background
+ writer</literal>.
+ </para>
+
+ <para>
+ <structname>pg_stat_io</structname> can be used to inform database tuning.
+ For example:
+ <itemizedlist>
+ <listitem>
+ <para>
+ A high <varname>evictions</varname> count can indicate that shared
+ buffers should be increased.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Client backends rely on the checkpointer to ensure data is persisted to
+ permanent storage. Large numbers of <varname>fsyncs</varname> by
+ <literal>client backend</literal>s could indicate a misconfiguration of
+ shared buffers or of the checkpointer. More information on configuring
+ the checkpointer can be found in <xref linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Normally, client backends should be able to rely on auxiliary processes
+ like the checkpointer and the background writer to write out dirty data
+ as much as possible. Large numbers of writes by client backends could
+ indicate a misconfiguration of shared buffers or of the checkpointer.
+ More information on configuring the checkpointer can be found in <xref
+ linkend="wal-configuration"/>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
</sect2>
--
2.34.1
[text/x-patch] v51-0001-Create-regress_tblspc-in-test_setup.patch (11.3K, ../../CAAKRu_af0YcV9CXG9M5R06tYRUzk13+H7qOsU6etmS4zE7RAUQ@mail.gmail.com/3-v51-0001-Create-regress_tblspc-in-test_setup.patch)
download | inline diff:
From bc2a91fbe68180d47179388b8303badcfdd5259c Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Thu, 19 Jan 2023 15:45:54 -0500
Subject: [PATCH v51 1/5] Create regress_tblspc in test_setup
Other tests may want to use a tablespace. Now that we have
allow_in_place_tablespaces, move the tablespace test to the end of the
parallel schedule and create the main tablespace it uses in test setup.
Modify the tablespace tests a bit to check for specific relations in the
test and not simply for the absence or presence of any objects in the
tablespace in case other tests leave objects around in the tablespace.
---
src/test/regress/expected/tablespace.out | 65 +++++++++++++-----------
src/test/regress/expected/test_setup.out | 3 ++
src/test/regress/parallel_schedule | 9 ++--
src/test/regress/sql/tablespace.sql | 42 ++++++++++-----
src/test/regress/sql/test_setup.sql | 4 ++
5 files changed, 75 insertions(+), 48 deletions(-)
diff --git a/src/test/regress/expected/tablespace.out b/src/test/regress/expected/tablespace.out
index c52cf1cfcf..007c00bfff 100644
--- a/src/test/regress/expected/tablespace.out
+++ b/src/test/regress/expected/tablespace.out
@@ -22,8 +22,6 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
--- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '';
-- This returns a relative path as of an effect of allow_in_place_tablespaces,
-- masking the tablespace OID used in the path name.
SELECT regexp_replace(pg_tablespace_location(oid), '(pg_tblspc)/(\d+)', '\1/NNN')
@@ -83,11 +81,14 @@ REINDEX (TABLESPACE regress_tblspace) INDEX regress_tblspace_test_tbl_idx;
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
ROLLBACK;
-- no relation moved to the new tablespace
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace';
- relname
----------
-(0 rows)
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
-- check that all indexes are moved to a new tablespace with different
-- relfilenode.
@@ -102,40 +103,46 @@ SELECT relfilenode as toast_filenode FROM pg_class
WHERE i.indrelid = c.reltoastrelid AND
c.relname = 'regress_tblspace_test_tbl') \gset
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
- relname
--------------------------------
- regress_tblspace_test_tbl_idx
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+ ?column?
+----------
+ t
(1 row)
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE regress_tblspace;
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
- relname
--------------------------------
- regress_tblspace_test_tbl_idx
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+ ?column?
+----------
+ t
(1 row)
-- Move back to the default tablespace.
ALTER INDEX regress_tblspace_test_tbl_idx SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
ORDER BY c.relname;
- relname
----------
-(0 rows)
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
REINDEX (TABLESPACE regress_tblspace, CONCURRENTLY) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
- relname
--------------------------------
- regress_tblspace_test_tbl_idx
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+ ?column?
+----------
+ t
(1 row)
SELECT relfilenode = :main_filenode AS main_same FROM pg_class
@@ -331,7 +338,7 @@ CREATE TABLE testschema.part1 PARTITION OF testschema.part FOR VALUES IN (1);
CREATE INDEX part_a_idx ON testschema.part (a) TABLESPACE regress_tblspace;
CREATE TABLE testschema.part2 PARTITION OF testschema.part FOR VALUES IN (2);
SELECT relname, spcname FROM pg_catalog.pg_tablespace t, pg_catalog.pg_class c
- where c.reltablespace = t.oid AND c.relname LIKE 'part%_idx';
+ where c.reltablespace = t.oid AND c.relname LIKE 'part%_idx' ORDER BY relname;
relname | spcname
-------------+------------------
part1_a_idx | regress_tblspace
diff --git a/src/test/regress/expected/test_setup.out b/src/test/regress/expected/test_setup.out
index 391b36d131..4f54fe20ec 100644
--- a/src/test/regress/expected/test_setup.out
+++ b/src/test/regress/expected/test_setup.out
@@ -18,6 +18,9 @@ SET synchronous_commit = on;
-- and most of the core regression tests still expect that.
--
GRANT ALL ON SCHEMA public TO public;
+-- Create a tablespace we can use in tests.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_tblspace LOCATION '';
--
-- These tables have traditionally been referenced by many tests,
-- so create and populate them. Insert only non-error values here.
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index a930dfe48c..15e015b3d6 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -11,11 +11,6 @@
# required setup steps
test: test_setup
-# run tablespace by itself, and early, because it forces a checkpoint;
-# we'd prefer not to have checkpoints later in the tests because that
-# interferes with crash-recovery testing.
-test: tablespace
-
# ----------
# The first group of parallel tests
# ----------
@@ -132,3 +127,7 @@ test: event_trigger oidjoins
# this test also uses event triggers, so likewise run it by itself
test: fast_default
+
+# run tablespace test at the end because it drops the tablespace created during
+# setup that other tests may use.
+test: tablespace
diff --git a/src/test/regress/sql/tablespace.sql b/src/test/regress/sql/tablespace.sql
index 21db433f2a..58a279e2f9 100644
--- a/src/test/regress/sql/tablespace.sql
+++ b/src/test/regress/sql/tablespace.sql
@@ -20,8 +20,6 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
--- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '';
-- This returns a relative path as of an effect of allow_in_place_tablespaces,
-- masking the tablespace OID used in the path name.
SELECT regexp_replace(pg_tablespace_location(oid), '(pg_tblspc)/(\d+)', '\1/NNN')
@@ -66,7 +64,9 @@ REINDEX (TABLESPACE regress_tblspace) INDEX regress_tblspace_test_tbl_idx;
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
ROLLBACK;
-- no relation moved to the new tablespace
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace';
-- check that all indexes are moved to a new tablespace with different
@@ -74,6 +74,7 @@ SELECT c.relname FROM pg_class c, pg_tablespace s
-- Save first the existing relfilenode for the toast and main relations.
SELECT relfilenode as main_filenode FROM pg_class
WHERE relname = 'regress_tblspace_test_tbl_idx' \gset
+
SELECT relfilenode as toast_filenode FROM pg_class
WHERE oid =
(SELECT i.indexrelid
@@ -81,24 +82,37 @@ SELECT relfilenode as toast_filenode FROM pg_class
pg_index i
WHERE i.indrelid = c.reltoastrelid AND
c.relname = 'regress_tblspace_test_tbl') \gset
+
REINDEX (TABLESPACE regress_tblspace) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
+
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE regress_tblspace;
ALTER TABLE regress_tblspace_test_tbl SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
+
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+
-- Move back to the default tablespace.
ALTER INDEX regress_tblspace_test_tbl_idx SET TABLESPACE pg_default;
-SELECT c.relname FROM pg_class c, pg_tablespace s
+SELECT c.relname <> 'regress_tblspace_test_tbl_idx',
+ c.relname <> 'regress_tblspace_test_tbl'
+ FROM pg_class c, pg_tablespace s
WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
ORDER BY c.relname;
+
REINDEX (TABLESPACE regress_tblspace, CONCURRENTLY) TABLE regress_tblspace_test_tbl;
-SELECT c.relname FROM pg_class c, pg_tablespace s
- WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace'
- ORDER BY c.relname;
+
+SELECT 'regress_tblspace_test_tbl_idx' IN
+ (SELECT c.relname
+ FROM pg_class c, pg_tablespace s
+ WHERE c.reltablespace = s.oid AND s.spcname = 'regress_tblspace');
+
SELECT relfilenode = :main_filenode AS main_same FROM pg_class
WHERE relname = 'regress_tblspace_test_tbl_idx';
SELECT relfilenode = :toast_filenode as toast_same FROM pg_class
@@ -225,7 +239,7 @@ CREATE TABLE testschema.part1 PARTITION OF testschema.part FOR VALUES IN (1);
CREATE INDEX part_a_idx ON testschema.part (a) TABLESPACE regress_tblspace;
CREATE TABLE testschema.part2 PARTITION OF testschema.part FOR VALUES IN (2);
SELECT relname, spcname FROM pg_catalog.pg_tablespace t, pg_catalog.pg_class c
- where c.reltablespace = t.oid AND c.relname LIKE 'part%_idx';
+ where c.reltablespace = t.oid AND c.relname LIKE 'part%_idx' ORDER BY relname;
\d testschema.part
\d+ testschema.part
\d testschema.part1
diff --git a/src/test/regress/sql/test_setup.sql b/src/test/regress/sql/test_setup.sql
index 02c0c84c3a..8439b38d21 100644
--- a/src/test/regress/sql/test_setup.sql
+++ b/src/test/regress/sql/test_setup.sql
@@ -23,6 +23,10 @@ SET synchronous_commit = on;
--
GRANT ALL ON SCHEMA public TO public;
+-- Create a tablespace we can use in tests.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_tblspace LOCATION '';
+
--
-- These tables have traditionally been referenced by many tests,
-- so create and populate them. Insert only non-error values here.
--
2.34.1
[text/x-patch] v51-0002-pgstat-Infrastructure-to-track-IO-operations.patch (28.6K, ../../CAAKRu_af0YcV9CXG9M5R06tYRUzk13+H7qOsU6etmS4zE7RAUQ@mail.gmail.com/4-v51-0002-pgstat-Infrastructure-to-track-IO-operations.patch)
download | inline diff:
From 1921e23c4650f497266c0f73114248a57e58778e Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 17 Jan 2023 16:10:34 -0500
Subject: [PATCH v51 2/5] pgstat: Infrastructure to track IO operations
Introduce "IOOp", an IO operation done by a backend, "IOObject", the
target object of the IO, and "IOContext", the context or location of the
IO operations on that object. For example, the checkpointer may write a
shared buffer out. This would be considered an IOOP_WRITE IOOp on an
IOOBJECT_RELATION IOObject in the IOCONTEXT_NORMAL IOContext by
BackendType B_CHECKPOINTER.
Each BackendType counts IOOps (evict, extend, fsync, read, reuse, and
write) per IOObject (relation, temp relation) per IOContext (normal,
bulkread, bulkwrite, or vacuum) through a call to pgstat_count_io_op().
Note that this commit introduces the infrastructure to count IO
Operation statistics. A subsequent commit will add calls to
pgstat_count_io_op() in the appropriate locations.
IOObject IOOBJECT_TEMP_RELATION concerns IO Operations on buffers
containing temporary table data, while IOObject IOOBJECT_RELATION
concerns IO Operations on buffers containing permanent relation data.
IOContext IOCONTEXT_NORMAL concerns operations on local and shared
buffers, while IOCONTEXT_BULKREAD, IOCONTEXT_BULKWRITE, and
IOCONTEXT_VACUUM IOContexts concern IO operations on buffers as part of
a BufferAccessStrategy.
Stats on IOOps on all IOObjects in all IOContexts for a given backend
are first counted in a backend's local memory and then flushed to shared
memory and accumulated with those from all other backends, exited and
live.
Some BackendTypes will not flush their pending statistics at regular
intervals and explicitly call pgstat_flush_io_ops() during the course of
normal operations to flush their backend-local IO operation statistics
to shared memory in a timely manner.
Because not all BackendType, IOObject, IOContext, IOOp combinations are
valid, the validity of the stats is checked before flushing pending
stats and before reading in the existing stats file to shared memory.
The aggregated stats in shared memory could be extended in the future
with per-backend stats -- useful for per connection IO statistics and
monitoring.
PGSTAT_FILE_FORMAT_ID should be bumped with this commit.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Reviewed-by: Kyotaro Horiguchi <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
doc/src/sgml/monitoring.sgml | 2 +
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 26 ++
src/backend/utils/activity/pgstat_bgwriter.c | 7 +-
.../utils/activity/pgstat_checkpointer.c | 7 +-
src/backend/utils/activity/pgstat_io.c | 386 ++++++++++++++++++
src/backend/utils/activity/pgstat_relation.c | 15 +-
src/backend/utils/activity/pgstat_shmem.c | 4 +
src/backend/utils/activity/pgstat_wal.c | 4 +-
src/backend/utils/adt/pgstatfuncs.c | 11 +-
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 68 +++
src/include/utils/pgstat_internal.h | 30 ++
src/tools/pgindent/typedefs.list | 6 +
15 files changed, 563 insertions(+), 7 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_io.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index e3a783abd0..c47d057a1d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5434,6 +5434,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
the <structname>pg_stat_bgwriter</structname>
view, <literal>archiver</literal> to reset all the counters shown in
the <structname>pg_stat_archiver</structname> view,
+ <literal>io</literal> to reset all the counters shown in the
+ <structname>pg_stat_io</structname> view,
<literal>wal</literal> to reset all the counters shown in the
<structname>pg_stat_wal</structname> view or
<literal>recovery_prefetch</literal> to reset all the counters shown
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index a80eda3cf4..7d7482dde0 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -22,6 +22,7 @@ OBJS = \
pgstat_checkpointer.o \
pgstat_database.o \
pgstat_function.o \
+ pgstat_io.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index a2b872c24b..518ee3f798 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -9,6 +9,7 @@ backend_sources += files(
'pgstat_checkpointer.c',
'pgstat_database.c',
'pgstat_function.c',
+ 'pgstat_io.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 0fa5370bcd..60fc4e761f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -72,6 +72,7 @@
* - pgstat_checkpointer.c
* - pgstat_database.c
* - pgstat_function.c
+ * - pgstat_io.c
* - pgstat_relation.c
* - pgstat_replslot.c
* - pgstat_slru.c
@@ -359,6 +360,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
+ [PGSTAT_KIND_IO] = {
+ .name = "io",
+
+ .fixed_amount = true,
+
+ .reset_all_cb = pgstat_io_reset_all_cb,
+ .snapshot_cb = pgstat_io_snapshot_cb,
+ },
+
[PGSTAT_KIND_SLRU] = {
.name = "slru",
@@ -582,6 +592,7 @@ pgstat_report_stat(bool force)
/* Don't expend a clock check if nothing to do */
if (dlist_is_empty(&pgStatPending) &&
+ !have_iostats &&
!have_slrustats &&
!pgstat_have_pending_wal())
{
@@ -628,6 +639,9 @@ pgstat_report_stat(bool force)
/* flush database / relation / function / ... stats */
partial_flush |= pgstat_flush_pending_entries(nowait);
+ /* flush IO stats */
+ partial_flush |= pgstat_flush_io(nowait);
+
/* flush wal stats */
partial_flush |= pgstat_flush_wal(nowait);
@@ -1322,6 +1336,12 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
+ /*
+ * Write IO stats struct
+ */
+ pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
+ write_chunk_s(fpout, &pgStatLocal.snapshot.io);
+
/*
* Write SLRU stats struct
*/
@@ -1496,6 +1516,12 @@ pgstat_read_statsfile(void)
if (!read_chunk_s(fpin, &shmem->checkpointer.stats))
goto error;
+ /*
+ * Read IO stats struct
+ */
+ if (!read_chunk_s(fpin, &shmem->io.stats))
+ goto error;
+
/*
* Read SLRU stats struct
*/
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 9247f2dda2..92be384b0d 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -24,7 +24,7 @@ PgStat_BgWriterStats PendingBgWriterStats = {0};
/*
- * Report bgwriter statistics
+ * Report bgwriter and IO statistics
*/
void
pgstat_report_bgwriter(void)
@@ -56,6 +56,11 @@ pgstat_report_bgwriter(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 3e9ab45103..26dec112f6 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -24,7 +24,7 @@ PgStat_CheckpointerStats PendingCheckpointerStats = {0};
/*
- * Report checkpointer statistics
+ * Report checkpointer and IO statistics
*/
void
pgstat_report_checkpointer(void)
@@ -62,6 +62,11 @@ pgstat_report_checkpointer(void)
* Clear out the statistics buffer, so it can be re-used.
*/
MemSet(&PendingCheckpointerStats, 0, sizeof(PendingCheckpointerStats));
+
+ /*
+ * Report IO statistics
+ */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
new file mode 100644
index 0000000000..b606f23eb8
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -0,0 +1,386 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_io.c
+ * Implementation of IO statistics.
+ *
+ * This file contains the implementation of IO statistics. It is kept separate
+ * from pgstat.c to enforce the line between the statistics access / storage
+ * implementation and the details about individual types of statistics.
+ *
+ * Copyright (c) 2021-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_io.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "utils/pgstat_internal.h"
+
+
+static PgStat_BktypeIO PendingIOStats;
+bool have_iostats = false;
+
+/*
+ * Check that stats have not been counted for any combination of IOObject,
+ * IOContext, and IOOp which are not tracked for the passed-in BackendType. The
+ * passed-in PgStat_BktypeIO must contain stats from the BackendType specified
+ * by the second parameter. Caller is responsible for locking the passed-in
+ * PgStat_BktypeIO, if needed.
+ */
+bool
+pgstat_bktype_io_stats_valid(PgStat_BktypeIO *backend_io,
+ BackendType bktype)
+{
+ bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
+
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ /*
+ * Don't bother trying to skip to the next loop iteration if
+ * pgstat_tracks_io_object() would return false here. We still
+ * need to validate that each counter is zero anyway.
+ */
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ /* No stats, so nothing to validate */
+ if (backend_io->data[io_object][io_context][io_op] == 0)
+ continue;
+
+ /* There are stats and there shouldn't be */
+ if (!bktype_tracked ||
+ !pgstat_tracks_io_op(bktype, io_object, io_context, io_op))
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
+void
+pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op)
+{
+ Assert(io_object < IOOBJECT_NUM_TYPES);
+ Assert(io_context < IOCONTEXT_NUM_TYPES);
+ Assert(io_op < IOOP_NUM_TYPES);
+ Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
+
+ PendingIOStats.data[io_object][io_context][io_op]++;
+
+ have_iostats = true;
+}
+
+PgStat_IO *
+pgstat_fetch_stat_io(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_IO);
+
+ return &pgStatLocal.snapshot.io;
+}
+
+/*
+ * Flush out locally pending IO statistics
+ *
+ * If no stats have been recorded, this function returns false.
+ *
+ * If nowait is true, this function returns true if the lock could not be
+ * acquired. Otherwise, return false.
+ */
+bool
+pgstat_flush_io(bool nowait)
+{
+ LWLock *bktype_lock;
+ PgStat_BktypeIO *bktype_shstats;
+
+ if (!have_iostats)
+ return false;
+
+ bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
+ bktype_shstats =
+ &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+
+ if (!nowait)
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
+ return true;
+
+ for (IOObject io_object = IOOBJECT_FIRST;
+ io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOOp io_op = IOOP_FIRST;
+ io_op < IOOP_NUM_TYPES; io_op++)
+ bktype_shstats->data[io_object][io_context][io_op] +=
+ PendingIOStats.data[io_object][io_context][io_op];
+
+ Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
+
+ LWLockRelease(bktype_lock);
+
+ memset(&PendingIOStats, 0, sizeof(PendingIOStats));
+
+ have_iostats = false;
+
+ return false;
+}
+
+const char *
+pgstat_get_io_context_name(IOContext io_context)
+{
+ switch (io_context)
+ {
+ case IOCONTEXT_BULKREAD:
+ return "bulkread";
+ case IOCONTEXT_BULKWRITE:
+ return "bulkwrite";
+ case IOCONTEXT_NORMAL:
+ return "normal";
+ case IOCONTEXT_VACUUM:
+ return "vacuum";
+ }
+
+ elog(ERROR, "unrecognized IOContext value: %d", io_context);
+ pg_unreachable();
+}
+
+const char *
+pgstat_get_io_object_name(IOObject io_object)
+{
+ switch (io_object)
+ {
+ case IOOBJECT_RELATION:
+ return "relation";
+ case IOOBJECT_TEMP_RELATION:
+ return "temp relation";
+ }
+
+ elog(ERROR, "unrecognized IOObject value: %d", io_object);
+ pg_unreachable();
+}
+
+void
+pgstat_io_reset_all_cb(TimestampTz ts)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+
+ memset(bktype_shstats, 0, sizeof(*bktype_shstats));
+ LWLockRelease(bktype_lock);
+ }
+}
+
+void
+pgstat_io_snapshot_cb(void)
+{
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ {
+ LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
+ PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+
+ LWLockAcquire(bktype_lock, LW_SHARED);
+
+ /*
+ * Use the lock in the first BackendType's PgStat_BktypeIO to protect
+ * the reset timestamp as well.
+ */
+ if (i == 0)
+ pgStatLocal.snapshot.io.stat_reset_timestamp =
+ pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+
+ /* using struct assignment due to better type safety */
+ *bktype_snap = *bktype_shstats;
+ LWLockRelease(bktype_lock);
+ }
+}
+
+/*
+* IO statistics are not collected for all BackendTypes.
+*
+* The following BackendTypes do not participate in the cumulative stats
+* subsystem or do not perform IO on which we currently track:
+* - Syslogger because it is not connected to shared memory
+* - Archiver because most relevant archiving IO is delegated to a
+* specialized command or module
+* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
+*
+* Function returns true if BackendType participates in the cumulative stats
+* subsystem for IO and false if it does not.
+*
+* When adding a new BackendType, also consider adding relevant restrictions to
+* pgstat_tracks_io_object() and pgstat_tracks_io_op().
+*/
+bool
+pgstat_tracks_io_bktype(BackendType bktype)
+{
+ /*
+ * List every type so that new backend types trigger a warning about
+ * needing to adjust this switch.
+ */
+ switch (bktype)
+ {
+ case B_INVALID:
+ case B_ARCHIVER:
+ case B_LOGGER:
+ case B_WAL_RECEIVER:
+ case B_WAL_WRITER:
+ return false;
+
+ case B_AUTOVAC_LAUNCHER:
+ case B_AUTOVAC_WORKER:
+ case B_BACKEND:
+ case B_BG_WORKER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_STANDALONE_BACKEND:
+ case B_STARTUP:
+ case B_WAL_SENDER:
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Some BackendTypes do not perform IO on certain IOObjects or in certain
+ * IOContexts. Some IOObjects are never operated on in some IOContexts. Check
+ * that the given BackendType is expected to do IO in the given IOContext and
+ * on the given IOObject and that the given IOObject is expected to be operated
+ * on in the given IOContext.
+ */
+bool
+pgstat_tracks_io_object(BackendType bktype, IOObject io_object,
+ IOContext io_context)
+{
+ bool no_temp_rel;
+
+ /*
+ * Some BackendTypes should never track IO statistics.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ return false;
+
+ /*
+ * Currently, IO on temporary relations can only occur in the
+ * IOCONTEXT_NORMAL IOContext.
+ */
+ if (io_context != IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * In core Postgres, only regular backends and WAL Sender processes
+ * executing queries will use local buffers and operate on temporary
+ * relations. Parallel workers will not use local buffers (see
+ * InitLocalBuffers()); however, extensions leveraging background workers
+ * have no such limitation, so track IO on IOOBJECT_TEMP_RELATION for
+ * BackendType B_BG_WORKER.
+ */
+ no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
+ bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
+
+ if (no_temp_rel && io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION)
+ return false;
+
+ /*
+ * Some BackendTypes do not currently perform any IO in certain
+ * IOContexts, and, while it may not be inherently incorrect for them to
+ * do so, excluding those rows from the view makes the view easier to use.
+ */
+ if ((bktype == B_CHECKPOINTER || bktype == B_BG_WRITER) &&
+ (io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE ||
+ io_context == IOCONTEXT_VACUUM))
+ return false;
+
+ if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
+ return false;
+
+ if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
+ io_context == IOCONTEXT_BULKWRITE)
+ return false;
+
+ return true;
+}
+
+/*
+ * Some BackendTypes will never do certain IOOps and some IOOps should not
+ * occur in certain IOContexts or on certain IOObjects. Check that the given
+ * IOOp is valid for the given BackendType in the given IOContext and on the
+ * given IOObject. Note that there are currently no cases of an IOOp being
+ * invalid for a particular BackendType only within a certain IOContext and/or
+ * only on a certain IOObject.
+ */
+bool
+pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
+ IOContext io_context, IOOp io_op)
+{
+ bool strategy_io_context;
+
+ /* if (io_context, io_object) will never collect stats, we're done */
+ if (!pgstat_tracks_io_object(bktype, io_object, io_context))
+ return false;
+
+ /*
+ * Some BackendTypes will not do certain IOOps.
+ */
+ if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) &&
+ (io_op == IOOP_READ || io_op == IOOP_EVICT))
+ return false;
+
+ if ((bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER) && io_op == IOOP_EXTEND)
+ return false;
+
+ /*
+ * Some IOOps are not valid in certain IOContexts and some IOOps are only
+ * valid in certain contexts.
+ */
+ if (io_context == IOCONTEXT_BULKREAD && io_op == IOOP_EXTEND)
+ return false;
+
+ strategy_io_context = io_context == IOCONTEXT_BULKREAD ||
+ io_context == IOCONTEXT_BULKWRITE || io_context == IOCONTEXT_VACUUM;
+
+ /*
+ * IOOP_REUSE is only relevant when a BufferAccessStrategy is in use.
+ */
+ if (!strategy_io_context && io_op == IOOP_REUSE)
+ return false;
+
+ /*
+ * IOOP_FSYNC IOOps done by a backend using a BufferAccessStrategy are
+ * counted in the IOCONTEXT_NORMAL IOContext. See comment in
+ * register_dirty_segment() for more details.
+ */
+ if (strategy_io_context && io_op == IOOP_FSYNC)
+ return false;
+
+ /*
+ * Temporary tables are not logged and thus do not require fsync'ing.
+ */
+ if (io_context == IOCONTEXT_NORMAL &&
+ io_object == IOOBJECT_TEMP_RELATION && io_op == IOOP_FSYNC)
+ return false;
+
+ return true;
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 2e20b93c20..f793ac1516 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -206,7 +206,7 @@ pgstat_drop_relation(Relation rel)
}
/*
- * Report that the table was just vacuumed.
+ * Report that the table was just vacuumed and flush IO statistics.
*/
void
pgstat_report_vacuum(Oid tableoid, bool shared,
@@ -258,10 +258,18 @@ pgstat_report_vacuum(Oid tableoid, bool shared,
}
pgstat_unlock_entry(entry_ref);
+
+ /*
+ * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
+ * however this will not be called until after an entire autovacuum cycle
+ * is done -- which will likely vacuum many relations -- or until the
+ * VACUUM command has processed all tables and committed.
+ */
+ pgstat_flush_io(false);
}
/*
- * Report that the table was just analyzed.
+ * Report that the table was just analyzed and flush IO statistics.
*
* Caller must provide new live- and dead-tuples estimates, as well as a
* flag indicating whether to reset the mod_since_analyze counter.
@@ -341,6 +349,9 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
+
+ /* see pgstat_report_vacuum() */
+ pgstat_flush_io(false);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index c1506b53d0..09fffd0e82 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -202,6 +202,10 @@ StatsShmemInit(void)
LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&ctl->io.locks[i],
+ LWTRANCHE_PGSTATS_DATA);
}
else
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index e7a82b5fed..e8598b2f4e 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -34,7 +34,7 @@ static WalUsage prevWalUsage;
/*
* Calculate how much WAL usage counters have increased and update
- * shared statistics.
+ * shared WAL and IO statistics.
*
* Must be called by processes that generate WAL, that do not call
* pgstat_report_stat(), like walwriter.
@@ -43,6 +43,8 @@ void
pgstat_report_wal(bool force)
{
pgstat_flush_wal(force);
+
+ pgstat_flush_io(force);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 6737493402..924698e6ae 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1587,7 +1587,12 @@ pg_stat_reset(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
-/* Reset some shared cluster-wide counters */
+/*
+ * Reset some shared cluster-wide counters
+ *
+ * When adding a new reset target, ideally the name should match that in
+ * pgstat_kind_infos, if relevant.
+ */
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
{
@@ -1604,6 +1609,8 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_BGWRITER);
pgstat_reset_of_kind(PGSTAT_KIND_CHECKPOINTER);
}
+ else if (strcmp(target, "io") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_IO);
else if (strcmp(target, "recovery_prefetch") == 0)
XLogPrefetchResetStats();
else if (strcmp(target, "wal") == 0)
@@ -1612,7 +1619,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"recovery_prefetch\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"io\", \"recovery_prefetch\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 96b3a1e1a0..c309e0233d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -332,6 +332,8 @@ typedef enum BackendType
B_WAL_WRITER,
} BackendType;
+#define BACKEND_NUM_TYPES (B_WAL_WRITER + 1)
+
extern PGDLLIMPORT BackendType MyBackendType;
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5e3326a3b9..9f09caa05f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -48,6 +48,7 @@ typedef enum PgStat_Kind
PGSTAT_KIND_ARCHIVER,
PGSTAT_KIND_BGWRITER,
PGSTAT_KIND_CHECKPOINTER,
+ PGSTAT_KIND_IO,
PGSTAT_KIND_SLRU,
PGSTAT_KIND_WAL,
} PgStat_Kind;
@@ -276,6 +277,55 @@ typedef struct PgStat_CheckpointerStats
PgStat_Counter buf_fsync_backend;
} PgStat_CheckpointerStats;
+
+/*
+ * Types related to counting IO operations
+ */
+typedef enum IOObject
+{
+ IOOBJECT_RELATION,
+ IOOBJECT_TEMP_RELATION,
+} IOObject;
+
+#define IOOBJECT_FIRST IOOBJECT_RELATION
+#define IOOBJECT_NUM_TYPES (IOOBJECT_TEMP_RELATION + 1)
+
+typedef enum IOContext
+{
+ IOCONTEXT_BULKREAD,
+ IOCONTEXT_BULKWRITE,
+ IOCONTEXT_NORMAL,
+ IOCONTEXT_VACUUM,
+} IOContext;
+
+#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
+#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
+
+typedef enum IOOp
+{
+ IOOP_EVICT,
+ IOOP_EXTEND,
+ IOOP_FSYNC,
+ IOOP_READ,
+ IOOP_REUSE,
+ IOOP_WRITE,
+} IOOp;
+
+#define IOOP_FIRST IOOP_EVICT
+#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
+
+typedef struct PgStat_BktypeIO
+{
+ PgStat_Counter data[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
+} PgStat_BktypeIO;
+
+typedef struct PgStat_IO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BktypeIO stats[BACKEND_NUM_TYPES];
+} PgStat_IO;
+
+
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -453,6 +503,24 @@ extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_bktype_io_stats_valid(PgStat_BktypeIO *context_ops,
+ BackendType bktype);
+extern void pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op);
+extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern const char *pgstat_get_io_context_name(IOContext io_context);
+extern const char *pgstat_get_io_object_name(IOObject io_object);
+
+extern bool pgstat_tracks_io_bktype(BackendType bktype);
+extern bool pgstat_tracks_io_object(BackendType bktype,
+ IOObject io_object, IOContext io_context);
+extern bool pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
+ IOContext io_context, IOOp io_op);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 12fd51f1ae..6badb2fde4 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -329,6 +329,17 @@ typedef struct PgStatShared_Checkpointer
PgStat_CheckpointerStats reset_offset;
} PgStatShared_Checkpointer;
+/* Shared-memory ready PgStat_IO */
+typedef struct PgStatShared_IO
+{
+ /*
+ * locks[i] protects stats.stats[i]. locks[0] also protects
+ * stats.stat_reset_timestamp.
+ */
+ LWLock locks[BACKEND_NUM_TYPES];
+ PgStat_IO stats;
+} PgStatShared_IO;
+
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -419,6 +430,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Archiver archiver;
PgStatShared_BgWriter bgwriter;
PgStatShared_Checkpointer checkpointer;
+ PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
} PgStat_ShmemControl;
@@ -442,6 +454,8 @@ typedef struct PgStat_Snapshot
PgStat_CheckpointerStats checkpointer;
+ PgStat_IO io;
+
PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS];
PgStat_WalStats wal;
@@ -549,6 +563,15 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+/*
+ * Functions in pgstat_io.c
+ */
+
+extern bool pgstat_flush_io(bool nowait);
+extern void pgstat_io_reset_all_cb(TimestampTz ts);
+extern void pgstat_io_snapshot_cb(void);
+
+
/*
* Functions in pgstat_relation.c
*/
@@ -643,6 +666,13 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid)
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
+/*
+ * Variables in pgstat_io.c
+ */
+
+extern PGDLLIMPORT bool have_iostats;
+
+
/*
* Variables in pgstat_slru.c
*/
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 09316039e4..65be0dea1b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1106,7 +1106,10 @@ ID
INFIX
INT128
INTERFACE_INFO
+IOContext
IOFuncSelector
+IOObject
+IOOp
IPCompareMethod
ITEM
IV
@@ -2015,6 +2018,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2024,6 +2028,7 @@ PgStat_ArchiverStats
PgStat_BackendFunctionEntry
PgStat_BackendSubEntry
PgStat_BgWriterStats
+PgStat_BktypeIO
PgStat_CheckpointerStats
PgStat_Counter
PgStat_EntryRef
@@ -2032,6 +2037,7 @@ PgStat_FetchConsistency
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
+PgStat_IO
PgStat_Kind
PgStat_KindInfo
PgStat_LocalState
--
2.34.1
[text/x-patch] v51-0003-pgstat-Count-IO-for-relations.patch (22.2K, ../../CAAKRu_af0YcV9CXG9M5R06tYRUzk13+H7qOsU6etmS4zE7RAUQ@mail.gmail.com/5-v51-0003-pgstat-Count-IO-for-relations.patch)
download | inline diff:
From 9ec02a2201ffc465be17fe0e9ee49d4e42a9d89f Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 17 Jan 2023 16:25:31 -0500
Subject: [PATCH v51 3/5] pgstat: Count IO for relations
Count IOOps done on IOObjects in IOContexts by various BackendTypes
using the IO stats infrastructure introduced by a previous commit.
The primary concern of these statistics is IO operations on data blocks
during the course of normal database operations. IO operations done by,
for example, the archiver or syslogger are not counted in these
statistics. WAL IO, temporary file IO, and IO done directly though smgr*
functions (such as when building an index) are not yet counted but would
be useful future additions.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
src/backend/storage/buffer/bufmgr.c | 110 ++++++++++++++++++++++----
src/backend/storage/buffer/freelist.c | 58 ++++++++++----
src/backend/storage/buffer/localbuf.c | 13 ++-
src/backend/storage/smgr/md.c | 24 ++++++
src/include/storage/buf_internals.h | 8 +-
src/include/storage/bufmgr.h | 7 +-
6 files changed, 184 insertions(+), 36 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8075828e8a..fffd846098 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -481,8 +481,9 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr);
-static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
+ bool *foundPtr, IOContext *io_context);
+static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
+ IOObject io_object, IOContext io_context);
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -823,6 +824,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BufferDesc *bufHdr;
Block bufBlock;
bool found;
+ IOContext io_context;
+ IOObject io_object;
bool isExtend;
bool isLocalBuf = SmgrIsTemp(smgr);
@@ -855,7 +858,14 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isLocalBuf)
{
- bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found);
+ /*
+ * LocalBufferAlloc() will set the io_context to IOCONTEXT_NORMAL. We
+ * do not use a BufferAccessStrategy for I/O of temporary tables.
+ * However, in some cases, the "strategy" may not be NULL, so we can't
+ * rely on IOContextForStrategy() to set the right IOContext for us.
+ * This may happen in cases like CREATE TEMPORARY TABLE AS...
+ */
+ bufHdr = LocalBufferAlloc(smgr, forkNum, blockNum, &found, &io_context);
if (found)
pgBufferUsage.local_blks_hit++;
else if (isExtend)
@@ -871,7 +881,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* not currently in memory.
*/
bufHdr = BufferAlloc(smgr, relpersistence, forkNum, blockNum,
- strategy, &found);
+ strategy, &found, &io_context);
if (found)
pgBufferUsage.shared_blks_hit++;
else if (isExtend)
@@ -986,7 +996,16 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
Assert(!(pg_atomic_read_u32(&bufHdr->state) & BM_VALID)); /* spinlock not needed */
- bufBlock = isLocalBuf ? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr);
+ if (isLocalBuf)
+ {
+ bufBlock = LocalBufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ bufBlock = BufHdrGetBlock(bufHdr);
+ io_object = IOOBJECT_RELATION;
+ }
if (isExtend)
{
@@ -995,6 +1014,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
/* don't set checksum for all-zero page */
smgrextend(smgr, forkNum, blockNum, (char *) bufBlock, false);
+ pgstat_count_io_op(io_object, io_context, IOOP_EXTEND);
+
/*
* NB: we're *not* doing a ScheduleBufferTagForWriteback here;
* although we're essentially performing a write. At least on linux
@@ -1020,6 +1041,8 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgrread(smgr, forkNum, blockNum, (char *) bufBlock);
+ pgstat_count_io_op(io_object, io_context, IOOP_READ);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -1113,14 +1136,19 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* *foundPtr is actually redundant with the buffer's BM_VALID flag, but
* we keep it for simplicity in ReadBuffer.
*
+ * io_context is passed as an output parameter to avoid calling
+ * IOContextForStrategy() when there is a shared buffers hit and no IO
+ * statistics need be captured.
+ *
* No locks are held either at entry or exit.
*/
static BufferDesc *
BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
BlockNumber blockNum,
BufferAccessStrategy strategy,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
+ bool from_ring;
BufferTag newTag; /* identity of requested block */
uint32 newHash; /* hash value for newTag */
LWLock *newPartitionLock; /* buffer partition lock for it */
@@ -1172,8 +1200,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* If we get here, previous attempts to read the buffer must
- * have failed ... but we shall bravely try again.
+ * have failed ... but we shall bravely try again. Set
+ * io_context since we will in fact need to count an IO
+ * Operation.
*/
+ *io_context = IOContextForStrategy(strategy);
*foundPtr = false;
}
}
@@ -1187,6 +1218,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*/
LWLockRelease(newPartitionLock);
+ *io_context = IOContextForStrategy(strategy);
+
/* Loop here in case we have to try another victim buffer */
for (;;)
{
@@ -1200,7 +1233,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* Select a victim buffer. The buffer is returned with its header
* spinlock still held!
*/
- buf = StrategyGetBuffer(strategy, &buf_state);
+ buf = StrategyGetBuffer(strategy, &buf_state, &from_ring);
Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 0);
@@ -1254,7 +1287,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
if (XLogNeedsFlush(lsn) &&
- StrategyRejectBuffer(strategy, buf))
+ StrategyRejectBuffer(strategy, buf, from_ring))
{
/* Drop lock/pin and loop around for another buffer */
LWLockRelease(BufferDescriptorGetContentLock(buf));
@@ -1269,7 +1302,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
smgr->smgr_rlocator.locator.dbOid,
smgr->smgr_rlocator.locator.relNumber);
- FlushBuffer(buf, NULL);
+ FlushBuffer(buf, NULL, IOOBJECT_RELATION, *io_context);
LWLockRelease(BufferDescriptorGetContentLock(buf));
ScheduleBufferTagForWriteback(&BackendWritebackContext,
@@ -1450,6 +1483,28 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
LWLockRelease(newPartitionLock);
+ if (oldFlags & BM_VALID)
+ {
+ /*
+ * When a BufferAccessStrategy is in use, blocks evicted from shared
+ * buffers are counted as IOOP_EVICT in the corresponding context
+ * (e.g. IOCONTEXT_BULKWRITE). Shared buffers are evicted by a
+ * strategy in two cases: 1) while initially claiming buffers for the
+ * strategy ring 2) to replace an existing strategy ring buffer
+ * because it is pinned or in use and cannot be reused.
+ *
+ * Blocks evicted from buffers already in the strategy ring are
+ * counted as IOOP_REUSE in the corresponding strategy context.
+ *
+ * At this point, we can accurately count evictions and reuses,
+ * because we have successfully claimed the valid buffer. Previously,
+ * we may have been forced to release the buffer due to concurrent
+ * pinners or erroring out.
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, *io_context,
+ from_ring ? IOOP_REUSE : IOOP_EVICT);
+ }
+
/*
* Buffer contents are currently invalid. Try to obtain the right to
* start I/O. If StartBufferIO returns false, then someone else managed
@@ -2570,7 +2625,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
@@ -2820,7 +2875,8 @@ BufferGetTag(Buffer buffer, RelFileLocator *rlocator, ForkNumber *forknum,
* as the second parameter. If not, pass NULL.
*/
static void
-FlushBuffer(BufferDesc *buf, SMgrRelation reln)
+FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
+ IOContext io_context)
{
XLogRecPtr recptr;
ErrorContextCallback errcallback;
@@ -2912,6 +2968,26 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
bufToWrite,
false);
+ /*
+ * When a strategy is in use, only flushes of dirty buffers already in the
+ * strategy ring are counted as strategy writes (IOCONTEXT
+ * [BULKREAD|BULKWRITE|VACUUM] IOOP_WRITE) for the purpose of IO
+ * statistics tracking.
+ *
+ * If a shared buffer initially added to the ring must be flushed before
+ * being used, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE.
+ *
+ * If a shared buffer which was added to the ring later because the
+ * current strategy buffer is pinned or in use or because all strategy
+ * buffers were dirty and rejected (for BAS_BULKREAD operations only)
+ * requires flushing, this is counted as an IOCONTEXT_NORMAL IOOP_WRITE
+ * (from_ring will be false).
+ *
+ * When a strategy is not in use, the write can only be a "regular" write
+ * of a dirty shared buffer (IOCONTEXT_NORMAL IOOP_WRITE).
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, io_context, IOOP_WRITE);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -3554,6 +3630,8 @@ FlushRelationBuffers(Relation rel)
buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_WRITE);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -3586,7 +3664,7 @@ FlushRelationBuffers(Relation rel)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, RelationGetSmgr(rel));
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3684,7 +3762,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, srelent->srel);
+ FlushBuffer(bufHdr, srelent->srel, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3894,7 +3972,7 @@ FlushDatabaseBuffers(Oid dbid)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr);
}
@@ -3921,7 +3999,7 @@ FlushOneBuffer(Buffer buffer)
Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr)));
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 7dec35801c..c690d5f15f 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,6 +15,7 @@
*/
#include "postgres.h"
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -81,12 +82,6 @@ typedef struct BufferAccessStrategyData
*/
int current;
- /*
- * True if the buffer just returned by StrategyGetBuffer had been in the
- * ring already.
- */
- bool current_was_in_ring;
-
/*
* Array of buffer numbers. InvalidBuffer (that is, zero) indicates we
* have not yet selected a buffer for this ring slot. For allocation
@@ -198,13 +193,15 @@ have_free_buffer(void)
* return the buffer with the buffer header spinlock still held.
*/
BufferDesc *
-StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
+StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring)
{
BufferDesc *buf;
int bgwprocno;
int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ *from_ring = false;
+
/*
* If given a strategy object, see whether it can select a buffer. We
* assume strategy objects don't need buffer_strategy_lock.
@@ -213,7 +210,10 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
{
buf = GetBufferFromRing(strategy, buf_state);
if (buf != NULL)
+ {
+ *from_ring = true;
return buf;
+ }
}
/*
@@ -602,7 +602,7 @@ FreeAccessStrategy(BufferAccessStrategy strategy)
/*
* GetBufferFromRing -- returns a buffer from the ring, or NULL if the
- * ring is empty.
+ * ring is empty / not usable.
*
* The bufhdr spin lock is held on the returned buffer.
*/
@@ -625,10 +625,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
*/
bufnum = strategy->buffers[strategy->current];
if (bufnum == InvalidBuffer)
- {
- strategy->current_was_in_ring = false;
return NULL;
- }
/*
* If the buffer is pinned we cannot use it under any circumstances.
@@ -644,7 +641,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) <= 1)
{
- strategy->current_was_in_ring = true;
*buf_state = local_buf_state;
return buf;
}
@@ -654,7 +650,6 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
* Tell caller to allocate a new buffer with the normal allocation
* strategy. He'll then replace this ring element via AddBufferToRing.
*/
- strategy->current_was_in_ring = false;
return NULL;
}
@@ -670,6 +665,39 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
strategy->buffers[strategy->current] = BufferDescriptorGetBuffer(buf);
}
+/*
+ * Utility function returning the IOContext of a given BufferAccessStrategy's
+ * strategy ring.
+ */
+IOContext
+IOContextForStrategy(BufferAccessStrategy strategy)
+{
+ if (!strategy)
+ return IOCONTEXT_NORMAL;
+
+ switch (strategy->btype)
+ {
+ case BAS_NORMAL:
+
+ /*
+ * Currently, GetAccessStrategy() returns NULL for
+ * BufferAccessStrategyType BAS_NORMAL, so this case is
+ * unreachable.
+ */
+ pg_unreachable();
+ return IOCONTEXT_NORMAL;
+ case BAS_BULKREAD:
+ return IOCONTEXT_BULKREAD;
+ case BAS_BULKWRITE:
+ return IOCONTEXT_BULKWRITE;
+ case BAS_VACUUM:
+ return IOCONTEXT_VACUUM;
+ }
+
+ elog(ERROR, "unrecognized BufferAccessStrategyType: %d", strategy->btype);
+ pg_unreachable();
+}
+
/*
* StrategyRejectBuffer -- consider rejecting a dirty buffer
*
@@ -682,14 +710,14 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
* if this buffer should be written and re-used.
*/
bool
-StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
+StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring)
{
/* We only do this in bulkread mode */
if (strategy->btype != BAS_BULKREAD)
return false;
/* Don't muck with behavior of normal buffer-replacement strategy */
- if (!strategy->current_was_in_ring ||
+ if (!from_ring ||
strategy->buffers[strategy->current] != BufferDescriptorGetBuffer(buf))
return false;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8372acc383..8e286db5df 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "catalog/catalog.h"
#include "executor/instrument.h"
+#include "pgstat.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "utils/guc_hooks.h"
@@ -107,7 +108,7 @@ PrefetchLocalBuffer(SMgrRelation smgr, ForkNumber forkNum,
*/
BufferDesc *
LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
- bool *foundPtr)
+ bool *foundPtr, IOContext *io_context)
{
BufferTag newTag; /* identity of requested block */
LocalBufferLookupEnt *hresult;
@@ -127,6 +128,14 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
+ /*
+ * IO Operations on local buffers are only done in IOCONTEXT_NORMAL. Set
+ * io_context here (instead of after a buffer hit would have returned) for
+ * convenience since we don't have to worry about the overhead of calling
+ * IOContextForStrategy().
+ */
+ *io_context = IOCONTEXT_NORMAL;
+
if (hresult)
{
b = hresult->id;
@@ -230,6 +239,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
buf_state &= ~BM_DIRTY;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_WRITE);
pgBufferUsage.local_blks_written++;
}
@@ -256,6 +266,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
ClearBufferTag(&bufHdr->tag);
buf_state &= ~(BM_VALID | BM_TAG_VALID);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_EVICT);
}
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 60c9905eff..8da813600c 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -983,6 +983,15 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
{
MdfdVec *v = &reln->md_seg_fds[forknum][segno - 1];
+ /*
+ * fsyncs done through mdimmedsync() should be tracked in a separate
+ * IOContext than those done through mdsyncfiletag() to differentiate
+ * between unavoidable client backend fsyncs (e.g. those done during
+ * index build) and those which ideally would have been done by the
+ * checkpointer. Since other IO operations bypassing the buffer
+ * manager could also be tracked in such an IOContext, wait until
+ * these are also tracked to track immediate fsyncs.
+ */
if (FileSync(v->mdfd_vfd, WAIT_EVENT_DATA_FILE_IMMEDIATE_SYNC) < 0)
ereport(data_sync_elevel(ERROR),
(errcode_for_file_access(),
@@ -1021,6 +1030,19 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
{
+ /*
+ * We have no way of knowing if the current IOContext is
+ * IOCONTEXT_NORMAL or IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] at this
+ * point, so count the fsync as being in the IOCONTEXT_NORMAL
+ * IOContext. This is probably okay, because the number of backend
+ * fsyncs doesn't say anything about the efficacy of the
+ * BufferAccessStrategy. And counting both fsyncs done in
+ * IOCONTEXT_NORMAL and IOCONTEXT_[BULKREAD, BULKWRITE, VACUUM] under
+ * IOCONTEXT_NORMAL is likely clearer when investigating the number of
+ * backend fsyncs.
+ */
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL, IOOP_FSYNC);
+
ereport(DEBUG1,
(errmsg_internal("could not forward fsync request because request queue is full")));
@@ -1410,6 +1432,8 @@ mdsyncfiletag(const FileTag *ftag, char *path)
if (need_to_close)
FileClose(file);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL, IOOP_FSYNC);
+
errno = save_errno;
return result;
}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index ed8aa2519c..0b44814740 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -15,6 +15,7 @@
#ifndef BUFMGR_INTERNALS_H
#define BUFMGR_INTERNALS_H
+#include "pgstat.h"
#include "port/atomics.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
@@ -391,11 +392,12 @@ extern void IssuePendingWritebacks(WritebackContext *context);
extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *tag);
/* freelist.c */
+extern IOContext IOContextForStrategy(BufferAccessStrategy bas);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
- uint32 *buf_state);
+ uint32 *buf_state, bool *from_ring);
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
- BufferDesc *buf);
+ BufferDesc *buf, bool from_ring);
extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
extern void StrategyNotifyBgWriter(int bgwprocno);
@@ -417,7 +419,7 @@ extern PrefetchBufferResult PrefetchLocalBuffer(SMgrRelation smgr,
ForkNumber forkNum,
BlockNumber blockNum);
extern BufferDesc *LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum,
- BlockNumber blockNum, bool *foundPtr);
+ BlockNumber blockNum, bool *foundPtr, IOContext *io_context);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 33eadbc129..b8a18b8081 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -23,7 +23,12 @@
typedef void *Block;
-/* Possible arguments for GetAccessStrategy() */
+/*
+ * Possible arguments for GetAccessStrategy().
+ *
+ * If adding a new BufferAccessStrategyType, also add a new IOContext so
+ * IO statistics using this strategy are tracked.
+ */
typedef enum BufferAccessStrategyType
{
BAS_NORMAL, /* Normal random access */
--
2.34.1
[text/x-patch] v51-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch (32.0K, ../../CAAKRu_af0YcV9CXG9M5R06tYRUzk13+H7qOsU6etmS4zE7RAUQ@mail.gmail.com/6-v51-0004-Add-system-view-tracking-IO-ops-per-backend-type.patch)
download | inline diff:
From cc70c22486f3d68c70e1059547a49179f007a51a Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Tue, 17 Jan 2023 16:28:27 -0500
Subject: [PATCH v51 4/5] Add system view tracking IO ops per backend type
Add pg_stat_io, a system view which tracks the number of IOOps
(evictions, reuses, reads, writes, extensions, and fsyncs) done on each
IOObject (relation, temp relation) in each IOContext ("normal" and those
using a BufferAccessStrategy) by each type of backend (e.g. client
backend, checkpointer).
Some BackendTypes do not accumulate IO operations statistics and will
not be included in the view.
Some IOObjects are never operated on in some IOContexts or by some
BackendTypes. These rows are omitted from the view. For example,
checkpointer will never operate on IOOBJECT_TEMP_RELATION data, so those
rows are omitted.
Some IOContexts are not used by some BackendTypes and will not be in the
view. For example, checkpointer does not use a BufferAccessStrategy
(currently), so there will be no rows for BufferAccessStrategy
IOContexts for checkpointer.
Some IOOps are invalid in combination with certain IOContexts and
certain IOObjects. Those cells will be NULL in the view to distinguish
between 0 observed IOOps of that type and an invalid combination. For
example, temporary tables are not fsynced so cells for all BackendTypes
for IOOBJECT_TEMP_RELATION and IOOP_FSYNC will be NULL.
Some BackendTypes never perform certain IOOps. Those cells will also be
NULL in the view. For example, bgwriter should not perform reads.
View stats are populated with statistics incremented when a backend
performs an IO Operation and maintained by the cumulative statistics
subsystem.
Each row of the view shows stats for a particular BackendType, IOObject,
IOContext combination (e.g. a client backend's operations on permanent
relations in shared buffers) and each column in the view is the total
number of IO Operations done (e.g. writes). So a cell in the view would
be, for example, the number of blocks of relation data written from
shared buffers by client backends since the last stats reset.
In anticipation of tracking WAL IO and non-block-oriented IO (such as
temporary file IO), the "op_bytes" column specifies the unit of the
"reads", "writes", and "extends" columns for a given row.
Note that some of the cells in the view are redundant with fields in
pg_stat_bgwriter (e.g. buffers_backend), however these have been kept in
pg_stat_bgwriter for backwards compatibility. Deriving the redundant
pg_stat_bgwriter stats from the IO operations stats structures was also
problematic due to the separate reset targets for 'bgwriter' and 'io'.
Suggested by Andres Freund
Catalog version should be bumped.
Author: Melanie Plageman <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de
---
contrib/amcheck/expected/check_heap.out | 34 ++++
contrib/amcheck/sql/check_heap.sql | 27 +++
src/backend/catalog/system_views.sql | 15 ++
src/backend/utils/adt/pgstatfuncs.c | 141 +++++++++++++++
src/include/catalog/pg_proc.dat | 9 +
src/test/regress/expected/rules.out | 12 ++
src/test/regress/expected/stats.out | 227 ++++++++++++++++++++++++
src/test/regress/sql/stats.sql | 141 +++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 607 insertions(+)
diff --git a/contrib/amcheck/expected/check_heap.out b/contrib/amcheck/expected/check_heap.out
index c010361025..e4785141a6 100644
--- a/contrib/amcheck/expected/check_heap.out
+++ b/contrib/amcheck/expected/check_heap.out
@@ -66,6 +66,22 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -88,6 +104,23 @@ SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock :=
-------+--------+--------+-----
(0 rows)
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+ ?column?
+----------
+ t
+(1 row)
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
SET ROLE regress_heaptest_role;
@@ -195,6 +228,7 @@ ERROR: cannot check relation "test_foreign_table"
DETAIL: This operation is not supported for foreign tables.
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/contrib/amcheck/sql/check_heap.sql b/contrib/amcheck/sql/check_heap.sql
index 298de6886a..6794ca4eb0 100644
--- a/contrib/amcheck/sql/check_heap.sql
+++ b/contrib/amcheck/sql/check_heap.sql
@@ -20,11 +20,29 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'NONE');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-FROZEN');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'ALL-VISIBLE');
+
-- Add some data so subsequent tests are not entirely trivial
INSERT INTO heaptest (a, b)
(SELECT gs, repeat('x', gs)
FROM generate_series(1,50) gs);
+-- pg_stat_io test:
+-- verify_heapam always uses a BAS_BULKREAD BufferAccessStrategy, whereas a
+-- sequential scan does so only if the table is large enough when compared to
+-- shared buffers (see initscan()). CREATE DATABASE ... also unconditionally
+-- uses a BAS_BULKREAD strategy, but we have chosen to use a tablespace and
+-- verify_heapam to provide coverage instead of adding another expensive
+-- operation to the main regression test suite.
+--
+-- Create an alternative tablespace and move the heaptest table to it, causing
+-- it to be rewritten and all the blocks to reliably evicted from shared
+-- buffers -- guaranteeing actual reads when we next select from it.
+SET allow_in_place_tablespaces = true;
+CREATE TABLESPACE regress_test_stats_tblspc LOCATION '';
+SELECT sum(reads) AS stats_bulkreads_before
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+ALTER TABLE heaptest SET TABLESPACE regress_test_stats_tblspc;
+
-- Check that valid options are not rejected nor corruption reported
-- for a non-empty table
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'none');
@@ -32,6 +50,14 @@ SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-frozen');
SELECT * FROM verify_heapam(relation := 'heaptest', skip := 'all-visible');
SELECT * FROM verify_heapam(relation := 'heaptest', startblock := 0, endblock := 0);
+-- verify_heapam should have read in the page written out by
+-- ALTER TABLE ... SET TABLESPACE ...
+-- causing an additional bulkread, which should be reflected in pg_stat_io.
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS stats_bulkreads_after
+ FROM pg_stat_io WHERE io_context = 'bulkread' \gset
+SELECT :stats_bulkreads_after > :stats_bulkreads_before;
+
CREATE ROLE regress_heaptest_role;
-- verify permissions are checked (error due to function not callable)
@@ -110,6 +136,7 @@ SELECT * FROM verify_heapam('test_foreign_table',
-- cleanup
DROP TABLE heaptest;
+DROP TABLESPACE regress_test_stats_tblspc;
DROP TABLE test_partition;
DROP TABLE test_partitioned;
DROP OWNED BY regress_heaptest_role; -- permissions
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..34ca0e739f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1117,6 +1117,21 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+CREATE VIEW pg_stat_io AS
+SELECT
+ b.backend_type,
+ b.io_object,
+ b.io_context,
+ b.reads,
+ b.writes,
+ b.extends,
+ b.op_bytes,
+ b.evictions,
+ b.reuses,
+ b.fsyncs,
+ b.stats_reset
+FROM pg_stat_get_io() b;
+
CREATE VIEW pg_stat_wal AS
SELECT
w.wal_records,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 924698e6ae..9d707c3521 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1245,6 +1245,147 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+* When adding a new column to the pg_stat_io view, add a new enum value
+* here above IO_NUM_COLUMNS.
+*/
+typedef enum io_stat_col
+{
+ IO_COL_BACKEND_TYPE,
+ IO_COL_IO_OBJECT,
+ IO_COL_IO_CONTEXT,
+ IO_COL_READS,
+ IO_COL_WRITES,
+ IO_COL_EXTENDS,
+ IO_COL_CONVERSION,
+ IO_COL_EVICTIONS,
+ IO_COL_REUSES,
+ IO_COL_FSYNCS,
+ IO_COL_RESET_TIME,
+ IO_NUM_COLUMNS,
+} io_stat_col;
+
+/*
+ * When adding a new IOOp, add a new io_stat_col and add a case to this
+ * function returning the corresponding io_stat_col.
+ */
+static io_stat_col
+pgstat_get_io_op_index(IOOp io_op)
+{
+ switch (io_op)
+ {
+ case IOOP_EVICT:
+ return IO_COL_EVICTIONS;
+ case IOOP_READ:
+ return IO_COL_READS;
+ case IOOP_REUSE:
+ return IO_COL_REUSES;
+ case IOOP_WRITE:
+ return IO_COL_WRITES;
+ case IOOP_EXTEND:
+ return IO_COL_EXTENDS;
+ case IOOP_FSYNC:
+ return IO_COL_FSYNCS;
+ }
+
+ elog(ERROR, "unrecognized IOOp value: %d", io_op);
+ pg_unreachable();
+}
+
+Datum
+pg_stat_get_io(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo;
+ PgStat_IO *backends_io_stats;
+ Datum reset_time;
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ backends_io_stats = pgstat_fetch_stat_io();
+
+ reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
+
+ for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ {
+ Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
+ PgStat_BktypeIO *bktype_stats = &backends_io_stats->stats[bktype];
+
+ /*
+ * In Assert builds, we can afford an extra loop through all of the
+ * counters checking that only expected stats are non-zero, since it
+ * keeps the non-Assert code cleaner.
+ */
+ Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype));
+
+ /*
+ * For those BackendTypes without IO Operation stats, skip
+ * representing them in the view altogether.
+ */
+ if (!pgstat_tracks_io_bktype(bktype))
+ continue;
+
+ for (IOObject io_obj = IOOBJECT_FIRST;
+ io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ {
+ const char *obj_name = pgstat_get_io_object_name(io_obj);
+
+ for (IOContext io_context = IOCONTEXT_FIRST;
+ io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ const char *context_name = pgstat_get_io_context_name(io_context);
+
+ Datum values[IO_NUM_COLUMNS] = {0};
+ bool nulls[IO_NUM_COLUMNS] = {0};
+
+ /*
+ * Some combinations of BackendType, IOObject, and IOContext
+ * are not valid for any type of IOOp. In such cases, omit the
+ * entire row from the view.
+ */
+ if (!pgstat_tracks_io_object(bktype, io_obj, io_context))
+ continue;
+
+ values[IO_COL_BACKEND_TYPE] = bktype_desc;
+ values[IO_COL_IO_CONTEXT] = CStringGetTextDatum(context_name);
+ values[IO_COL_IO_OBJECT] = CStringGetTextDatum(obj_name);
+ values[IO_COL_RESET_TIME] = TimestampTzGetDatum(reset_time);
+
+ /*
+ * Hard-code this to the value of BLCKSZ for now. Future
+ * values could include XLOG_BLCKSZ, once WAL IO is tracked,
+ * and constant multipliers, once non-block-oriented IO (e.g.
+ * temporary file IO) is tracked.
+ */
+ values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
+
+ for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ int col_idx = pgstat_get_io_op_index(io_op);
+
+ /*
+ * Some combinations of BackendType and IOOp, of IOContext
+ * and IOOp, and of IOObject and IOOp are not tracked. Set
+ * these cells in the view NULL.
+ */
+ nulls[col_idx] = !pgstat_tracks_io_op(bktype, io_obj, io_context, io_op);
+
+ if (nulls[col_idx])
+ continue;
+
+ values[col_idx] =
+ Int64GetDatum(bktype_stats->data[io_obj][io_context][io_op]);
+ }
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ }
+ }
+
+ return (Datum) 0;
+}
+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 86eb8e8c58..2e804c5bd4 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5690,6 +5690,15 @@
proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
+{ oid => '8459', descr => 'statistics: per backend type IO statistics',
+ proname => 'pg_stat_get_io', provolatile => 'v',
+ prorows => '30', proretset => 't',
+ proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,text,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{backend_type,io_object,io_context,reads,writes,extends,op_bytes,evictions,reuses,fsyncs,stats_reset}',
+ prosrc => 'pg_stat_get_io' },
+
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e7a2f5856a..174b725fff 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1876,6 +1876,18 @@ pg_stat_gssapi| SELECT pid,
gss_enc AS encrypted
FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
WHERE (client_port IS NOT NULL);
+pg_stat_io| SELECT backend_type,
+ io_object,
+ io_context,
+ reads,
+ writes,
+ extends,
+ op_bytes,
+ evictions,
+ reuses,
+ fsyncs,
+ stats_reset
+ FROM pg_stat_get_io() b(backend_type, io_object, io_context, reads, writes, extends, op_bytes, evictions, reuses, fsyncs, stats_reset);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 1d84407a03..3ad38da0dd 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1126,4 +1126,231 @@ SELECT pg_stat_get_subscription_stats(NULL);
(1 row)
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+ ?column?
+----------
+ t
+(1 row)
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_tblspace;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+ count
+-------
+ 100
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+ count
+-------
+ 5000
+(1 row)
+
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+ ?column? | ?column? | ?column? | ?column?
+----------+----------+----------+----------
+ t | t | t | t
+(1 row)
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_tblspace;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+ ?column?
+----------
+ t
+(1 row)
+
+RESET temp_buffers;
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+ ?column? | ?column?
+----------+----------
+ t | t
+(1 row)
+
+RESET wal_skip_threshold;
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+ pg_stat_have_stats
+--------------------
+ t
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+ ?column?
+----------
+ t
+(1 row)
+
-- End of Stats Test
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index b4d6753c71..5badd09a1c 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -536,4 +536,145 @@ SELECT pg_stat_get_replication_slot(NULL);
SELECT pg_stat_get_subscription_stats(NULL);
+-- Test that the following operations are tracked in pg_stat_io:
+-- - reads of target blocks into shared buffers
+-- - writes of shared buffers to permanent storage
+-- - extends of relations using shared buffers
+-- - fsyncs done to ensure the durability of data dirtying shared buffers
+
+-- There is no test for blocks evicted from shared buffers, because we cannot
+-- be sure of the state of shared buffers at the point the test is run.
+
+-- Create a regular table and insert some data to generate IOCONTEXT_NORMAL
+-- extends.
+SELECT sum(extends) AS io_sum_shared_before_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+CREATE TABLE test_io_shared(a int);
+INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_shared_after_extends
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_extends > :io_sum_shared_before_extends;
+
+-- After a checkpoint, there should be some additional IOCONTEXT_NORMAL writes
+-- and fsyncs.
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_before_
+-- See comment above for rationale for two explicit CHECKPOINTs.
+CHECKPOINT;
+CHECKPOINT;
+SELECT sum(writes) AS writes, sum(fsyncs) AS fsyncs
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'relation' \gset io_sum_shared_after_
+
+SELECT :io_sum_shared_after_writes > :io_sum_shared_before_writes;
+SELECT current_setting('fsync') = 'off'
+ OR :io_sum_shared_after_fsyncs > :io_sum_shared_before_fsyncs;
+
+-- Change the tablespace so that the table is rewritten directly, then SELECT
+-- from it to cause it to be read back into shared buffers.
+SELECT sum(reads) AS io_sum_shared_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+ALTER TABLE test_io_shared SET TABLESPACE regress_tblspace;
+-- SELECT from the table so that the data is read into shared buffers and
+-- io_context 'normal', io_object 'relation' reads are counted.
+SELECT COUNT(*) FROM test_io_shared;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reads) AS io_sum_shared_after_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
+SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
+
+-- Test that the follow IOCONTEXT_LOCAL IOOps are tracked in pg_stat_io:
+-- - eviction of local buffers in order to reuse them
+-- - reads of temporary table blocks into local buffers
+-- - writes of local buffers to permanent storage
+-- - extends of temporary tables
+
+-- Set temp_buffers to its minimum so that we can trigger writes with fewer
+-- inserted tuples. Do so in a new session in case temporary tables have been
+-- accessed by previous tests in this session.
+\c
+SET temp_buffers TO 100;
+CREATE TEMPORARY TABLE test_io_local(a int, b TEXT);
+SELECT sum(extends) AS extends, sum(evictions) AS evictions, sum(writes) AS writes
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_before_
+-- Insert tuples into the temporary table, generating extends in the stats.
+-- Insert enough values that we need to reuse and write out dirty local
+-- buffers, generating evictions and writes.
+INSERT INTO test_io_local SELECT generate_series(1, 5000) as id, repeat('a', 200);
+-- Ensure the table is large enough to exceed our temp_buffers setting.
+SELECT pg_relation_size('test_io_local') / current_setting('block_size')::int8 > 100;
+
+SELECT sum(reads) AS io_sum_local_before_reads
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+-- Read in evicted buffers, generating reads.
+SELECT COUNT(*) FROM test_io_local;
+SELECT pg_stat_force_next_flush();
+SELECT sum(evictions) AS evictions,
+ sum(reads) AS reads,
+ sum(writes) AS writes,
+ sum(extends) AS extends
+ FROM pg_stat_io
+ WHERE io_context = 'normal' AND io_object = 'temp relation' \gset io_sum_local_after_
+SELECT :io_sum_local_after_evictions > :io_sum_local_before_evictions,
+ :io_sum_local_after_reads > :io_sum_local_before_reads,
+ :io_sum_local_after_writes > :io_sum_local_before_writes,
+ :io_sum_local_after_extends > :io_sum_local_before_extends;
+
+-- Change the tablespaces so that the temporary table is rewritten to other
+-- local buffers, exercising a different codepath than standard local buffer
+-- writes.
+ALTER TABLE test_io_local SET TABLESPACE regress_tblspace;
+SELECT pg_stat_force_next_flush();
+SELECT sum(writes) AS io_sum_local_new_tblspc_writes
+ FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'temp relation' \gset
+SELECT :io_sum_local_new_tblspc_writes > :io_sum_local_after_writes;
+RESET temp_buffers;
+
+-- Test that reuse of strategy buffers and reads of blocks into these reused
+-- buffers while VACUUMing are tracked in pg_stat_io.
+
+-- Set wal_skip_threshold smaller than the expected size of
+-- test_io_vac_strategy so that, even if wal_level is minimal, VACUUM FULL will
+-- fsync the newly rewritten test_io_vac_strategy instead of writing it to WAL.
+-- Writing it to WAL will result in the newly written relation pages being in
+-- shared buffers -- preventing us from testing BAS_VACUUM BufferAccessStrategy
+-- reads.
+SET wal_skip_threshold = '1 kB';
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_before_
+CREATE TABLE test_io_vac_strategy(a int, b int) WITH (autovacuum_enabled = 'false');
+INSERT INTO test_io_vac_strategy SELECT i, i from generate_series(1, 8000)i;
+-- Ensure that the next VACUUM will need to perform IO by rewriting the table
+-- first with VACUUM (FULL).
+VACUUM (FULL) test_io_vac_strategy;
+VACUUM (PARALLEL 0) test_io_vac_strategy;
+SELECT pg_stat_force_next_flush();
+SELECT sum(reuses) AS reuses, sum(reads) AS reads
+ FROM pg_stat_io WHERE io_context = 'vacuum' \gset io_sum_vac_strategy_after_
+SELECT :io_sum_vac_strategy_after_reads > :io_sum_vac_strategy_before_reads,
+ :io_sum_vac_strategy_after_reuses > :io_sum_vac_strategy_before_reuses;
+RESET wal_skip_threshold;
+
+-- Test that extends done by a CTAS, which uses a BAS_BULKWRITE
+-- BufferAccessStrategy, are tracked in pg_stat_io.
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_before
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+CREATE TABLE test_io_bulkwrite_strategy AS SELECT i FROM generate_series(1,100)i;
+SELECT pg_stat_force_next_flush();
+SELECT sum(extends) AS io_sum_bulkwrite_strategy_extends_after
+ FROM pg_stat_io WHERE io_context = 'bulkwrite' \gset
+SELECT :io_sum_bulkwrite_strategy_extends_after > :io_sum_bulkwrite_strategy_extends_before;
+
+-- Test IO stats reset
+SELECT pg_stat_have_stats('io', 0, 0);
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_pre_reset
+ FROM pg_stat_io \gset
+SELECT pg_stat_reset_shared('io');
+SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) AS io_stats_post_reset
+ FROM pg_stat_io \gset
+SELECT :io_stats_post_reset < :io_stats_pre_reset;
+
-- End of Stats Test
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 65be0dea1b..970a0cfd1d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3376,6 +3376,7 @@ intset_internal_node
intset_leaf_node
intset_node
intvKEY
+io_stat_col
itemIdCompact
itemIdCompactData
iterator
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-24 08:22 ` Kyotaro Horiguchi <[email protected]>
2023-01-24 08:25 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2023-01-24 08:22 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Hello.
At Thu, 19 Jan 2023 21:15:34 -0500, Melanie Plageman <[email protected]> wrote in
> Oh dear-- an extra FlushBuffer() snuck in there somehow.
> Removed it in attached v51.
> Also, I fixed an issue in my tablespace.sql updates
I only looked 0002 and 0004.
(Sorry for the random order of the comment..)
0002:
+ Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
This is relatively complex checking. We already asserts-out increments
of invalid counters. Thus this is checking if some unrelated codes
clobbered them, which we do only when consistency is critical. Is
there any needs to do that here? I saw another occurance of the same
assertion.
-/* Reset some shared cluster-wide counters */
+/*
+ * Reset some shared cluster-wide counters
+ *
+ * When adding a new reset target, ideally the name should match that in
+ * pgstat_kind_infos, if relevant.
+ */
I'm not sure the addition is useful..
+pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op)
+{
+ Assert(io_object < IOOBJECT_NUM_TYPES);
+ Assert(io_context < IOCONTEXT_NUM_TYPES);
+ Assert(io_op < IOOP_NUM_TYPES);
+ Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
Is there any reason for not checking the value ranges at the
bottom-most functions? They can lead to out-of-bounds access so I
don't think we need to continue execution for such invalid values.
+ no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
+ bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
+ bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
I'm not sure I like to omit parentheses for such a long Boolean
expression on the right side.
+ write_chunk_s(fpout, &pgStatLocal.snapshot.io);
+ if (!read_chunk_s(fpin, &shmem->io.stats))
The names of the functions hardly make sense alone to me. How about
write_struct()/read_struct()? (I personally prefer to use
write_chunk() directly..)
+ PgStat_BktypeIO
This patch abbreviates "backend" as "bk" but "be" is used in many
places. I think that naming should follow the predecessors.
0004:
system_views.sql:
+FROM pg_stat_get_io() b;
What does the "b" stand for? (Backend? then "s" or "i" seems
straight-forward.)
+ nulls[col_idx] = !pgstat_tracks_io_op(bktype, io_obj, io_context, io_op);
+
+ if (nulls[col_idx])
+ continue;
+
+ values[col_idx] =
+ Int64GetDatum(bktype_stats->data[io_obj][io_context][io_op]);
This is a bit hard to read since it requires to follow the condition
flow. The following is simpler and I thhink close to our standard.
if (pgstat_tacks_io_op())
values[col_idx] =
Int64GetDatum(bktype_stats->data[io_obj][io_context][io_op]);
else
nulls[col_idx] = true;
> + Number of read operations in units of <varname>op_bytes</varname>.
I may be the only one who see the name as umbiguous between "total
number of handled bytes" and "bytes hadled at an operation". Can't it
be op_blocksize or just block_size?
+ b.io_object,
+ b.io_context,
It's uncertain to me why only the two columns are prefixed by
"io". Don't "object_type" and just "context" work instead?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
@ 2023-01-24 08:25 ` Kyotaro Horiguchi <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2023-01-24 08:25 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
At Tue, 24 Jan 2023 17:22:03 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> +pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op)
> +{
> + Assert(io_object < IOOBJECT_NUM_TYPES);
> + Assert(io_context < IOCONTEXT_NUM_TYPES);
> + Assert(io_op < IOOP_NUM_TYPES);
> + Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
>
> Is there any reason for not checking the value ranges at the
> bottom-most functions? They can lead to out-of-bounds access so I
To make sure, the "They" means "out-of-range io_object/context/op
values"..
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
@ 2023-01-24 22:35 ` Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Andres Freund @ 2023-01-24 22:35 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Hi,
On 2023-01-24 17:22:03 +0900, Kyotaro Horiguchi wrote:
> Hello.
>
> At Thu, 19 Jan 2023 21:15:34 -0500, Melanie Plageman <[email protected]> wrote in
> > Oh dear-- an extra FlushBuffer() snuck in there somehow.
> > Removed it in attached v51.
> > Also, I fixed an issue in my tablespace.sql updates
>
> I only looked 0002 and 0004.
> (Sorry for the random order of the comment..)
>
> 0002:
>
> + Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
>
> This is relatively complex checking. We already asserts-out increments
> of invalid counters. Thus this is checking if some unrelated codes
> clobbered them, which we do only when consistency is critical. Is
> there any needs to do that here? I saw another occurance of the same
> assertion.
I found it useful to find problems.
> + no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
> + bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
> + bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
>
> I'm not sure I like to omit parentheses for such a long Boolean
> expression on the right side.
What parens would help?
> + write_chunk_s(fpout, &pgStatLocal.snapshot.io);
> + if (!read_chunk_s(fpin, &shmem->io.stats))
>
> The names of the functions hardly make sense alone to me. How about
> write_struct()/read_struct()? (I personally prefer to use
> write_chunk() directly..)
That's not related to this patch - there's several existing callers for
it. And write_struct wouldn't be better imo, because it's not just for
structs.
> + PgStat_BktypeIO
>
> This patch abbreviates "backend" as "bk" but "be" is used in many
> places. I think that naming should follow the predecessors.
The precedence aren't consistent unfortunately :)
> > + Number of read operations in units of <varname>op_bytes</varname>.
>
> I may be the only one who see the name as umbiguous between "total
> number of handled bytes" and "bytes hadled at an operation". Can't it
> be op_blocksize or just block_size?
>
> + b.io_object,
> + b.io_context,
No, block wouldn't be helpful - we'd like to use this for something that isn't
uniform blocks.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-01-25 07:56 ` Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Kyotaro Horiguchi @ 2023-01-25 07:56 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
At Tue, 24 Jan 2023 14:35:12 -0800, Andres Freund <[email protected]> wrote in
> > 0002:
> >
> > + Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
> >
> > This is relatively complex checking. We already asserts-out increments
> > of invalid counters. Thus this is checking if some unrelated codes
> > clobbered them, which we do only when consistency is critical. Is
> > there any needs to do that here? I saw another occurance of the same
> > assertion.
>
> I found it useful to find problems.
Okay.
> > + no_temp_rel = bktype == B_AUTOVAC_LAUNCHER || bktype == B_BG_WRITER ||
> > + bktype == B_CHECKPOINTER || bktype == B_AUTOVAC_WORKER ||
> > + bktype == B_STANDALONE_BACKEND || bktype == B_STARTUP;
> >
> > I'm not sure I like to omit parentheses for such a long Boolean
> > expression on the right side.
>
> What parens would help?
I thought about the following.
no_temp_rel =
(bktype == B_AUTOVAC_LAUNCHER ||
bktype == B_BG_WRITER ||
bktype == B_CHECKPOINTER ||
bktype == B_AUTOVAC_WORKER ||
bktype == B_STANDALONE_BACKEND ||
bktype == B_STARTUP);
> > + write_chunk_s(fpout, &pgStatLocal.snapshot.io);
> > + if (!read_chunk_s(fpin, &shmem->io.stats))
> >
> > The names of the functions hardly make sense alone to me. How about
> > write_struct()/read_struct()? (I personally prefer to use
> > write_chunk() directly..)
>
> That's not related to this patch - there's several existing callers for
> it. And write_struct wouldn't be better imo, because it's not just for
> structs.
Hmm. Then what the "_s" stands for?
> > + PgStat_BktypeIO
> >
> > This patch abbreviates "backend" as "bk" but "be" is used in many
> > places. I think that naming should follow the predecessors.
>
> The precedence aren't consistent unfortunately :)
Uuuummmmm. Okay, just I like "be" there! Anyway, I don't strongly
push that.
> > > + Number of read operations in units of <varname>op_bytes</varname>.
> >
> > I may be the only one who see the name as umbiguous between "total
> > number of handled bytes" and "bytes hadled at an operation". Can't it
> > be op_blocksize or just block_size?
> >
> > + b.io_object,
> > + b.io_context,
>
> No, block wouldn't be helpful - we'd like to use this for something that isn't
> uniform blocks.
What does the field show in that case? The mean of operation size? Or
one row per opration size? If the former, the name looks somewhat
wrong. If the latter, block_size seems making sense.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
@ 2023-02-08 06:38 ` Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Andres Freund @ 2023-02-08 06:38 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Hi,
I did another read through the series. I do have some minor changes, but
they're minor. I think this is ready for commit. I plan to start pushing
tomorrow.
The changes I made are:
- the tablespace test changes didn't quite work in isolation / needed a bit of
polishing
- moved the tablespace changes to later in the series
- split the tests out of the commit adding the view into its own commit
- minor code formatting things (e.g. didn't like nested for()s without {})
On 2023-01-25 16:56:17 +0900, Kyotaro Horiguchi wrote:
> At Tue, 24 Jan 2023 14:35:12 -0800, Andres Freund <[email protected]> wrote in
> > > + write_chunk_s(fpout, &pgStatLocal.snapshot.io);
> > > + if (!read_chunk_s(fpin, &shmem->io.stats))
> > >
> > > The names of the functions hardly make sense alone to me. How about
> > > write_struct()/read_struct()? (I personally prefer to use
> > > write_chunk() directly..)
> >
> > That's not related to this patch - there's several existing callers for
> > it. And write_struct wouldn't be better imo, because it's not just for
> > structs.
>
> Hmm. Then what the "_s" stands for?
Size. It's a macro that just forwards to read_chunk()/write_chunk().
> > > > + Number of read operations in units of <varname>op_bytes</varname>.
> > >
> > > I may be the only one who see the name as umbiguous between "total
> > > number of handled bytes" and "bytes hadled at an operation". Can't it
> > > be op_blocksize or just block_size?
> > >
> > > + b.io_object,
> > > + b.io_context,
> >
> > No, block wouldn't be helpful - we'd like to use this for something that isn't
> > uniform blocks.
>
> What does the field show in that case? The mean of operation size? Or
> one row per opration size? If the former, the name looks somewhat
> wrong. If the latter, block_size seems making sense.
1, so that it's clear that the rest are in bytes.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-02-09 05:03 ` Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: Andres Freund @ 2023-02-09 05:03 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Hi,
On 2023-02-07 22:38:14 -0800, Andres Freund wrote:
> I did another read through the series. I do have some minor changes, but
> they're minor. I think this is ready for commit. I plan to start pushing
> tomorrow.
Pushed the first (and biggest) commit. More tomorrow.
Already can't wait to see incremental improvements of this version of
pg_stat_io ;). Tracking buffer hits. Tracking Wal IO. Tracking relation IO
bypassing shared buffers. Per connection IO statistics. Tracking IO time.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-02-11 18:24 ` Andres Freund <[email protected]>
2023-02-14 19:08 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-22 01:50 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-03-04 23:21 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
1 sibling, 3 replies; 36+ messages in thread
From: Andres Freund @ 2023-02-11 18:24 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Hi,
On 2023-02-08 21:03:19 -0800, Andres Freund wrote:
> Pushed the first (and biggest) commit. More tomorrow.
Just pushed the actual pg_stat_io view, the splitting of the tablespace test,
and the pg_stat_io tests.
Yay!
Thanks all for patch and review!
> Already can't wait to see incremental improvements of this version of
> pg_stat_io ;). Tracking buffer hits. Tracking Wal IO. Tracking relation IO
> bypassing shared buffers. Per connection IO statistics. Tracking IO time.
That's still the case.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-02-14 19:08 ` Andres Freund <[email protected]>
2 siblings, 0 replies; 36+ messages in thread
From: Andres Freund @ 2023-02-14 19:08 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Hi,
On 2023-02-11 10:24:37 -0800, Andres Freund wrote:
> Just pushed the actual pg_stat_io view, the splitting of the tablespace test,
> and the pg_stat_io tests.
One thing I started to wonder about since is whether we should remove the io_
prefix from io_object, io_context. The prefixes make sense on the C level, but
it's not clear to me that that's also the case on the table level.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-02-22 01:50 ` Justin Pryzby <[email protected]>
2023-02-22 04:09 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2 siblings, 1 reply; 36+ messages in thread
From: Justin Pryzby @ 2023-02-22 01:50 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
On Sat, Feb 11, 2023 at 10:24:37AM -0800, Andres Freund wrote:
> On 2023-02-08 21:03:19 -0800, Andres Freund wrote:
> > Pushed the first (and biggest) commit. More tomorrow.
>
> Just pushed the actual pg_stat_io view, the splitting of the tablespace test,
> and the pg_stat_io tests.
pg_stat_io says:
* Some BackendTypes do not currently perform any IO in certain
* IOContexts, and, while it may not be inherently incorrect for them to
* do so, excluding those rows from the view makes the view easier to use.
if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
return false;
if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
io_context == IOCONTEXT_BULKWRITE)
return false;
What about these combinations? Aren't these also "can't happen" ?
relation | bulkread | autovacuum worker
relation | bulkread | autovacuum launcher
relation | vacuum | startup
--
Justin
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-22 01:50 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
@ 2023-02-22 04:09 ` Justin Pryzby <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Justin Pryzby @ 2023-02-22 04:09 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
On Tue, Feb 21, 2023 at 07:50:35PM -0600, Justin Pryzby wrote:
> On Sat, Feb 11, 2023 at 10:24:37AM -0800, Andres Freund wrote:
> > On 2023-02-08 21:03:19 -0800, Andres Freund wrote:
> > > Pushed the first (and biggest) commit. More tomorrow.
> >
> > Just pushed the actual pg_stat_io view, the splitting of the tablespace test,
> > and the pg_stat_io tests.
>
> pg_stat_io says:
>
> * Some BackendTypes do not currently perform any IO in certain
> * IOContexts, and, while it may not be inherently incorrect for them to
> * do so, excluding those rows from the view makes the view easier to use.
>
> if (bktype == B_AUTOVAC_LAUNCHER && io_context == IOCONTEXT_VACUUM)
> return false;
>
> if ((bktype == B_AUTOVAC_WORKER || bktype == B_AUTOVAC_LAUNCHER) &&
> io_context == IOCONTEXT_BULKWRITE)
> return false;
>
> What about these combinations? Aren't these also "can't happen" ?
>
> relation | bulkread | autovacuum worker
> relation | bulkread | autovacuum launcher
> relation | vacuum | startup
Nevermind - at least these are possible.
(gdb) p MyBackendType
$1 = B_AUTOVAC_WORKER
(gdb) p io_object
$2 = IOOBJECT_RELATION
(gdb) p io_context
$3 = IOCONTEXT_BULKREAD
(gdb) p io_op
$4 = IOOP_EVICT
(gdb) bt
...
#9 0x0000557b2f6097a3 in ReadBufferExtended (reln=0x7ff5ccee36b8, forkNum=forkNum@entry=MAIN_FORKNUM, blockNum=blockNum@entry=16, mode=mode@entry=RBM_NORMAL, strategy=0x557b305fb568) at ../src/include/utils/rel.h:573
#10 0x0000557b2f3057c0 in heapgetpage (sscan=sscan@entry=0x557b305fb158, block=block@entry=16) at ../src/backend/access/heap/heapam.c:405
#11 0x0000557b2f305d6c in heapgettup_pagemode (scan=scan@entry=0x557b305fb158, dir=dir@entry=ForwardScanDirection, nkeys=0, key=0x0) at ../src/backend/access/heap/heapam.c:885
#12 0x0000557b2f306956 in heap_getnext (sscan=sscan@entry=0x557b305fb158, direction=direction@entry=ForwardScanDirection) at ../src/backend/access/heap/heapam.c:1122
#13 0x0000557b2f59be0c in do_autovacuum () at ../src/backend/postmaster/autovacuum.c:2061
#14 0x0000557b2f59ccf7 in AutoVacWorkerMain (argc=argc@entry=0, argv=argv@entry=0x0) at ../src/backend/postmaster/autovacuum.c:1716
#15 0x0000557b2f59cdd8 in StartAutoVacWorker () at ../src/backend/postmaster/autovacuum.c:1494
#16 0x0000557b2f5a561a in StartAutovacuumWorker () at ../src/backend/postmaster/postmaster.c:5481
#17 0x0000557b2f5a5a39 in process_pm_pmsignal () at ../src/backend/postmaster/postmaster.c:5192
#18 0x0000557b2f5a5d7e in ServerLoop () at ../src/backend/postmaster/postmaster.c:1770
#19 0x0000557b2f5a73da in PostmasterMain (argc=9, argv=<optimized out>) at ../src/backend/postmaster/postmaster.c:1463
#20 0x0000557b2f4dfc39 in main (argc=9, argv=0x557b30568f50) at ../src/backend/main/main.c:200
--
Justin
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-03-04 23:21 ` Tom Lane <[email protected]>
2023-03-06 06:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2 siblings, 1 reply; 36+ messages in thread
From: Tom Lane @ 2023-03-04 23:21 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Andres Freund <[email protected]> writes:
> Just pushed the actual pg_stat_io view, the splitting of the tablespace test,
> and the pg_stat_io tests.
One of the test cases is flapping a bit:
diff -U3 /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/expected/stats.out /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/results/stats.out
--- /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/expected/stats.out 2023-03-04 21:30:05.891579466 +0100
+++ /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/results/stats.out 2023-03-04 21:34:26.745552661 +0100
@@ -1201,7 +1201,7 @@
SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
?column?
----------
- t
+ f
(1 row)
DROP TABLE test_io_shared;
There are two instances of this today [1][2], and I've seen it before
but failed to note down where.
regards, tom lane
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=grison&dt=2023-03-04%2021%3A19%3A39
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mule&dt=2023-03-04%2020%3A30%3A05
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-03-04 23:21 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
@ 2023-03-06 06:24 ` Kyotaro Horiguchi <[email protected]>
2023-03-06 06:48 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Kyotaro Horiguchi @ 2023-03-06 06:24 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
At Sat, 04 Mar 2023 18:21:09 -0500, Tom Lane <[email protected]> wrote in
> Andres Freund <[email protected]> writes:
> > Just pushed the actual pg_stat_io view, the splitting of the tablespace test,
> > and the pg_stat_io tests.
>
> One of the test cases is flapping a bit:
>
> diff -U3 /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/expected/stats.out /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/results/stats.out
> --- /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/expected/stats.out 2023-03-04 21:30:05.891579466 +0100
> +++ /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/results/stats.out 2023-03-04 21:34:26.745552661 +0100
> @@ -1201,7 +1201,7 @@
> SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
> ?column?
> ----------
> - t
> + f
> (1 row)
>
> DROP TABLE test_io_shared;
>
> There are two instances of this today [1][2], and I've seen it before
> but failed to note down where.
The concurrent autoanalyze below is logged as performing at least one
page read from the table. It is unclear, however, how that analyze
operation resulted in 19 hits and 2 reads on the (I think) single-page
relation.
In any case, I think we need to avoid such concurrent autovacuum/analyze.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=grison&dt=2023-03-04%2021%3A19%3A39
2023-03-04 22:36:27.781 CET [4073:106] pg_regress/stats LOG: statement: ALTER TABLE test_io_shared SET TABLESPACE regress_tblspace;
2023-03-04 22:36:27.838 CET [4073:107] pg_regress/stats LOG: statement: SELECT COUNT(*) FROM test_io_shared;
2023-03-04 22:36:27.864 CET [4255:5] LOG: automatic analyze of table "regression.public.test_io_shared"
avg read rate: 5.208 MB/s, avg write rate: 5.208 MB/s
buffer usage: 17 hits, 2 misses, 2 dirtied
2023-03-04 22:36:28.024 CET [4073:108] pg_regress/stats LOG: statement: SELECT pg_stat_force_next_flush();
2023-03-04 22:36:28.024 CET [4073:108] pg_regress/stats LOG: statement: SELECT pg_stat_force_next_flush();
2023-03-04 22:36:28.027 CET [4073:109] pg_regress/stats LOG: statement: SELECT sum(reads) AS io_sum_shared_after_reads
FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation'
> [1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=grison&dt=2023-03-04%2021%3A19%3A39
> [2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mule&dt=2023-03-04%2020%3A30%3A05
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-03-04 23:21 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-03-06 06:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
@ 2023-03-06 06:48 ` Kyotaro Horiguchi <[email protected]>
2023-03-06 15:09 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Kyotaro Horiguchi @ 2023-03-06 06:48 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
At Mon, 06 Mar 2023 15:24:25 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> In any case, I think we need to avoid such concurrent autovacuum/analyze.
If it is correct, I believe the attached fix works.
regads.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
[text/x-patch] fix_stats_test.diff (1.3K, ../../[email protected]/2-fix_stats_test.diff)
download | inline diff:
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 937b2101b3..023ec5ecc4 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1137,7 +1137,7 @@ SELECT pg_stat_get_subscription_stats(NULL);
-- extends.
SELECT sum(extends) AS io_sum_shared_before_extends
FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
-CREATE TABLE test_io_shared(a int);
+CREATE TABLE test_io_shared(a int) WITH (autovacuum_enabled = 'false');
INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
SELECT pg_stat_force_next_flush();
pg_stat_force_next_flush
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 74e592aa8a..aa6552befd 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -549,7 +549,7 @@ SELECT pg_stat_get_subscription_stats(NULL);
-- extends.
SELECT sum(extends) AS io_sum_shared_before_extends
FROM pg_stat_io WHERE io_context = 'normal' AND io_object = 'relation' \gset
-CREATE TABLE test_io_shared(a int);
+CREATE TABLE test_io_shared(a int) WITH (autovacuum_enabled = 'false');
INSERT INTO test_io_shared SELECT i FROM generate_series(1,100)i;
SELECT pg_stat_force_next_flush();
SELECT sum(extends) AS io_sum_shared_after_extends
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-03-04 23:21 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-03-06 06:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-03-06 06:48 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
@ 2023-03-06 15:09 ` Melanie Plageman <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Melanie Plageman @ 2023-03-06 15:09 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
On Mon, Mar 6, 2023 at 1:48 AM Kyotaro Horiguchi
<[email protected]> wrote:
>
> At Mon, 06 Mar 2023 15:24:25 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> > In any case, I think we need to avoid such concurrent autovacuum/analyze.
>
> If it is correct, I believe the attached fix works.
Thanks for investigating this!
Yes, this fix looks correct and makes sense to me.
On Mon, Mar 6, 2023 at 1:24 AM Kyotaro Horiguchi
<[email protected]> wrote:
>
> At Sat, 04 Mar 2023 18:21:09 -0500, Tom Lane <[email protected]> wrote in
> > Andres Freund <[email protected]> writes:
> > > Just pushed the actual pg_stat_io view, the splitting of the tablespace test,
> > > and the pg_stat_io tests.
> >
> > One of the test cases is flapping a bit:
> >
> > diff -U3 /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/expected/stats.out /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/results/stats.out
> > --- /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/expected/stats.out 2023-03-04 21:30:05.891579466 +0100
> > +++ /home/pg/build-farm-15/buildroot/HEAD/pgsql.build/src/test/regress/results/stats.out 2023-03-04 21:34:26.745552661 +0100
> > @@ -1201,7 +1201,7 @@
> > SELECT :io_sum_shared_after_reads > :io_sum_shared_before_reads;
> > ?column?
> > ----------
> > - t
> > + f
> > (1 row)
> >
> > DROP TABLE test_io_shared;
> >
> > There are two instances of this today [1][2], and I've seen it before
> > but failed to note down where.
>
> The concurrent autoanalyze below is logged as performing at least one
> page read from the table. It is unclear, however, how that analyze
> operation resulted in 19 hits and 2 reads on the (I think) single-page
> relation.
Yes, it is a single page.
I think there could be a few different reasons by it is 2 misses/2
dirtied, but the one that seems most likely is that I/O of other
relations done during this autovac/analyze of this relation is counted
in the same global variables (like catalog tables).
- Melanie
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
@ 2023-02-26 18:20 ` Tom Lane <[email protected]>
2023-02-26 18:52 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Tom Lane @ 2023-02-26 18:20 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Andres Freund <[email protected]> writes:
> Pushed the first (and biggest) commit. More tomorrow.
I hadn't run my buildfarm-compile-warning scraper for a little while,
but I just did, and I find that this commit is causing warnings on
no fewer than 14 buildfarm animals. They all look like
ayu | 2023-02-25 23:02:08 | pgstat_io.c:40:14: warning: comparison of constant 2 with expression of type 'IOObject' (aka 'enum IOObject') is always true [-Wtautological-constant-out-of-range-compare]
ayu | 2023-02-25 23:02:08 | pgstat_io.c:43:16: warning: comparison of constant 4 with expression of type 'IOContext' (aka 'enum IOContext') is always true [-Wtautological-constant-out-of-range-compare]
ayu | 2023-02-25 23:02:08 | pgstat_io.c:70:19: warning: comparison of constant 2 with expression of type 'IOObject' (aka 'enum IOObject') is always true [-Wtautological-constant-out-of-range-compare]
ayu | 2023-02-25 23:02:08 | pgstat_io.c:71:20: warning: comparison of constant 4 with expression of type 'IOContext' (aka 'enum IOContext') is always true [-Wtautological-constant-out-of-range-compare]
ayu | 2023-02-25 23:02:08 | pgstat_io.c:115:14: warning: comparison of constant 2 with expression of type 'IOObject' (aka 'enum IOObject') is always true [-Wtautological-constant-out-of-range-compare]
ayu | 2023-02-25 23:02:08 | pgstat_io.c:118:16: warning: comparison of constant 4 with expression of type 'IOContext' (aka 'enum IOContext') is always true [-Wtautological-constant-out-of-range-compare]
ayu | 2023-02-25 23:02:08 | pgstatfuncs.c:1329:12: warning: comparison of constant 2 with expression of type 'IOObject' (aka 'enum IOObject') is always true [-Wtautological-constant-out-of-range-compare]
ayu | 2023-02-25 23:02:08 | pgstatfuncs.c:1334:17: warning: comparison of constant 4 with expression of type 'IOContext' (aka 'enum IOContext') is always true [-Wtautological-constant-out-of-range-compare]
That is, these compilers think that comparisons like
io_object < IOOBJECT_NUM_TYPES
io_context < IOCONTEXT_NUM_TYPES
are constant-true. This seems not good; if they were to actually
act on this observation, by removing those loop-ending tests,
we'd have a problem.
The issue seems to be that code like this:
typedef enum IOContext
{
IOCONTEXT_BULKREAD,
IOCONTEXT_BULKWRITE,
IOCONTEXT_NORMAL,
IOCONTEXT_VACUUM,
} IOContext;
#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
is far too cute for its own good. I'm not sure about how to fix it
either. I thought of defining
#define IOCONTEXT_LAST IOCONTEXT_VACUUM
and make the loop conditions like "io_context <= IOCONTEXT_LAST",
but that doesn't actually fix the problem.
(Even aside from that, I do not find this coding even a little bit
mistake-proof: you still have to remember to update the #define
when adding another enum value.)
We have similar code involving enum ForkNumber but it looks to me
like the loop variables are always declared as plain "int". That
might be the path of least resistance here.
regards, tom lane
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
@ 2023-02-26 18:52 ` Tom Lane <[email protected]>
2023-02-27 14:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Tom Lane @ 2023-02-26 18:52 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
I wrote:
> The issue seems to be that code like this:
> ...
> is far too cute for its own good.
Oh, there's another thing here that qualifies as too-cute: loops like
for (IOObject io_object = IOOBJECT_FIRST;
io_object < IOOBJECT_NUM_TYPES; io_object++)
make it look like we could define these enums as 1-based rather
than 0-based, but if we did this code would fail, because it's
confusing "the number of values" with "1 more than the last value".
Again, we could fix that with tests like "io_context <= IOCONTEXT_LAST",
but I don't see the point of adding more macros rather than removing
some. We do need IOOBJECT_NUM_TYPES to declare array sizes with,
so I think we should nuke the "xxx_FIRST" macros as being not worth
the electrons they're written on, and write these loops like
for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
which is not actually adding any assumptions that you don't already
make by using io_object as a C array subscript.
regards, tom lane
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-26 18:52 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
@ 2023-02-27 14:24 ` Melanie Plageman <[email protected]>
2023-02-27 15:30 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Melanie Plageman @ 2023-02-27 14:24 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
On Sun, Feb 26, 2023 at 1:52 PM Tom Lane <[email protected]> wrote:
>
> I wrote:
> > The issue seems to be that code like this:
> > ...
> > is far too cute for its own good.
>
> Oh, there's another thing here that qualifies as too-cute: loops like
>
> for (IOObject io_object = IOOBJECT_FIRST;
> io_object < IOOBJECT_NUM_TYPES; io_object++)
>
> make it look like we could define these enums as 1-based rather
> than 0-based, but if we did this code would fail, because it's
> confusing "the number of values" with "1 more than the last value".
>
> Again, we could fix that with tests like "io_context <= IOCONTEXT_LAST",
> but I don't see the point of adding more macros rather than removing
> some. We do need IOOBJECT_NUM_TYPES to declare array sizes with,
> so I think we should nuke the "xxx_FIRST" macros as being not worth
> the electrons they're written on, and write these loops like
>
> for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
>
> which is not actually adding any assumptions that you don't already
> make by using io_object as a C array subscript.
Attached is a patch to remove the *_FIRST macros.
I was going to add in code to change
for (IOObject io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
to
for (IOObject io_object = 0; (int) io_object < IOOBJECT_NUM_TYPES;
io_object++)
but then I couldn't remember why we didn't just do
for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
I recall that when passing that loop variable into a function I was
getting a compiler warning that required me to cast the value back to an
enum to silence it:
pgstat_tracks_io_op(bktype, (IOObject) io_object,
io_context, io_op))
However, I am now unable to reproduce that warning.
Moreover, I see in cases like table_block_relation_size() with
ForkNumber, the variable i is passed with no cast to smgrnblocks().
- Melanie
Attachments:
[text/x-patch] v1-0001-Remove-potentially-misleading-_FIRST-macros.patch (4.9K, ../../CAAKRu_Yci1Pw5qnoYdPz-FFqX-qpS0y0R_TA=rA_njACGp-SGQ@mail.gmail.com/2-v1-0001-Remove-potentially-misleading-_FIRST-macros.patch)
download | inline diff:
From cce6dc75e9e4fc9adc018a1d05874be5f3be96ae Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 27 Feb 2023 08:22:53 -0500
Subject: [PATCH v1 1/2] Remove potentially misleading *_FIRST macros
28e626bde00ef introduced IO statistic enums IOOp, IOObject, and
IOContext along with macros *_FIRST intended for use when looping
through the enumerated values of each. Per discussion in [1] these
macros are confusing and error-prone. Remove them.
[1] https://www.postgresql.org/message-id/23770.1677437567%40sss.pgh.pa.us
---
src/backend/utils/activity/pgstat_io.c | 17 ++++++-----------
src/backend/utils/adt/pgstatfuncs.c | 10 ++++------
src/include/pgstat.h | 3 ---
3 files changed, 10 insertions(+), 20 deletions(-)
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 0e07e0848d..c478b126fa 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -36,18 +36,16 @@ pgstat_bktype_io_stats_valid(PgStat_BktypeIO *backend_io,
{
bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
- for (IOObject io_object = IOOBJECT_FIRST;
- io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOObject io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
{
- for (IOContext io_context = IOCONTEXT_FIRST;
- io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOContext io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
{
/*
* Don't bother trying to skip to the next loop iteration if
* pgstat_tracks_io_object() would return false here. We still
* need to validate that each counter is zero anyway.
*/
- for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ for (IOOp io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
{
/* No stats, so nothing to validate */
if (backend_io->data[io_object][io_context][io_op] == 0)
@@ -111,14 +109,11 @@ pgstat_flush_io(bool nowait)
else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
return true;
- for (IOObject io_object = IOOBJECT_FIRST;
- io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (IOObject io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
{
- for (IOContext io_context = IOCONTEXT_FIRST;
- io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOContext io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
{
- for (IOOp io_op = IOOP_FIRST;
- io_op < IOOP_NUM_TYPES; io_op++)
+ for (IOOp io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
bktype_shstats->data[io_object][io_context][io_op] +=
PendingIOStats.data[io_object][io_context][io_op];
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 9d707c3521..12eda4ade0 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1306,7 +1306,7 @@ pg_stat_get_io(PG_FUNCTION_ARGS)
reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
- for (BackendType bktype = B_INVALID; bktype < BACKEND_NUM_TYPES; bktype++)
+ for (BackendType bktype = 0; bktype < BACKEND_NUM_TYPES; bktype++)
{
Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
PgStat_BktypeIO *bktype_stats = &backends_io_stats->stats[bktype];
@@ -1325,13 +1325,11 @@ pg_stat_get_io(PG_FUNCTION_ARGS)
if (!pgstat_tracks_io_bktype(bktype))
continue;
- for (IOObject io_obj = IOOBJECT_FIRST;
- io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ for (IOObject io_obj = 0; io_obj < IOOBJECT_NUM_TYPES; io_obj++)
{
const char *obj_name = pgstat_get_io_object_name(io_obj);
- for (IOContext io_context = IOCONTEXT_FIRST;
- io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (IOContext io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
{
const char *context_name = pgstat_get_io_context_name(io_context);
@@ -1359,7 +1357,7 @@ pg_stat_get_io(PG_FUNCTION_ARGS)
*/
values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
- for (IOOp io_op = IOOP_FIRST; io_op < IOOP_NUM_TYPES; io_op++)
+ for (IOOp io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
{
int col_idx = pgstat_get_io_op_index(io_op);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index db9675884f..f43fac09ed 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -287,7 +287,6 @@ typedef enum IOObject
IOOBJECT_TEMP_RELATION,
} IOObject;
-#define IOOBJECT_FIRST IOOBJECT_RELATION
#define IOOBJECT_NUM_TYPES (IOOBJECT_TEMP_RELATION + 1)
typedef enum IOContext
@@ -298,7 +297,6 @@ typedef enum IOContext
IOCONTEXT_VACUUM,
} IOContext;
-#define IOCONTEXT_FIRST IOCONTEXT_BULKREAD
#define IOCONTEXT_NUM_TYPES (IOCONTEXT_VACUUM + 1)
typedef enum IOOp
@@ -311,7 +309,6 @@ typedef enum IOOp
IOOP_WRITE,
} IOOp;
-#define IOOP_FIRST IOOP_EVICT
#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
typedef struct PgStat_BktypeIO
--
2.37.2
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-26 18:52 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 14:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-02-27 15:30 ` Tom Lane <[email protected]>
2023-02-27 19:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Tom Lane @ 2023-02-27 15:30 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Andres Freund <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Melanie Plageman <[email protected]> writes:
> Attached is a patch to remove the *_FIRST macros.
> I was going to add in code to change
> for (IOObject io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
> to
> for (IOObject io_object = 0; (int) io_object < IOOBJECT_NUM_TYPES; io_object++)
I don't really like that proposal. ISTM it's just silencing the
messenger rather than addressing the underlying problem, namely that
there's no guarantee that an IOObject variable can hold the value
IOOBJECT_NUM_TYPES, which it had better do if you want the loop to
terminate. Admittedly it's quite unlikely that these three enums would
grow to the point that that becomes an actual hazard for them --- but
IMO it's still bad practice and a bad precedent for future code.
> but then I couldn't remember why we didn't just do
> for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
> I recall that when passing that loop variable into a function I was
> getting a compiler warning that required me to cast the value back to an
> enum to silence it:
> pgstat_tracks_io_op(bktype, (IOObject) io_object,
> io_context, io_op))
> However, I am now unable to reproduce that warning.
> Moreover, I see in cases like table_block_relation_size() with
> ForkNumber, the variable i is passed with no cast to smgrnblocks().
Yeah, my druthers would be to just do it the way we do comparable
things with ForkNumber. I don't feel like we need to invent a
better way here.
The risk of needing to cast when using the "int" loop variable
as an enum is obviously the downside of that approach, but we have
not seen any indication that any compilers actually do warn.
It's interesting that you did see such a warning ... I wonder which
compiler you were using at the time?
regards, tom lane
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-26 18:52 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 14:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-02-27 15:30 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
@ 2023-02-27 19:03 ` Melanie Plageman <[email protected]>
2023-02-27 19:58 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Melanie Plageman @ 2023-02-27 19:03 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
On Mon, Feb 27, 2023 at 10:30 AM Tom Lane <[email protected]> wrote:
>
> Melanie Plageman <[email protected]> writes:
> > Attached is a patch to remove the *_FIRST macros.
> > I was going to add in code to change
>
> > for (IOObject io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
> > to
> > for (IOObject io_object = 0; (int) io_object < IOOBJECT_NUM_TYPES; io_object++)
>
> I don't really like that proposal. ISTM it's just silencing the
> messenger rather than addressing the underlying problem, namely that
> there's no guarantee that an IOObject variable can hold the value
> IOOBJECT_NUM_TYPES, which it had better do if you want the loop to
> terminate. Admittedly it's quite unlikely that these three enums would
> grow to the point that that becomes an actual hazard for them --- but
> IMO it's still bad practice and a bad precedent for future code.
That's fair. Patch attached.
> > but then I couldn't remember why we didn't just do
>
> > for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
>
> > I recall that when passing that loop variable into a function I was
> > getting a compiler warning that required me to cast the value back to an
> > enum to silence it:
>
> > pgstat_tracks_io_op(bktype, (IOObject) io_object,
> > io_context, io_op))
>
> > However, I am now unable to reproduce that warning.
> > Moreover, I see in cases like table_block_relation_size() with
> > ForkNumber, the variable i is passed with no cast to smgrnblocks().
>
> Yeah, my druthers would be to just do it the way we do comparable
> things with ForkNumber. I don't feel like we need to invent a
> better way here.
>
> The risk of needing to cast when using the "int" loop variable
> as an enum is obviously the downside of that approach, but we have
> not seen any indication that any compilers actually do warn.
> It's interesting that you did see such a warning ... I wonder which
> compiler you were using at the time?
so, pretty much any version of clang I tried with
-Wsign-conversion produces a warning.
<source>:35:32: warning: implicit conversion changes signedness: 'int'
to 'IOOp' (aka 'enum IOOp') [-Wsign-conversion]
I didn't do the casts in the attached patch since they aren't done elsewhere.
- Melanie
Attachments:
[text/x-patch] Change-IO-stats-enum-loop-variables-to-ints.patch (4.0K, ../../CAAKRu_bLrO9Wk4_Nuq1CpJ45RUohaq_51Lty28BF2TwjXyTd+Q@mail.gmail.com/2-Change-IO-stats-enum-loop-variables-to-ints.patch)
download | inline diff:
From 34fee4a3d1d1353aa38a95b3afc2d302a5f586ff Mon Sep 17 00:00:00 2001
From: Melanie Plageman <[email protected]>
Date: Mon, 27 Feb 2023 08:48:11 -0500
Subject: [PATCH v1 2/2] Change IO stats enum loop variables to ints
Per [1], using an enum as the loop variable with a macro-defined
termination condition of #_enum_values + 1 is not guaranteed to be safe
- as compilers are free to make enums as small as a char.
Some (older) compilers will notice this when building with
-Wtautological-constant-out-of-range-compare.
[1] https://www.postgresql.org/message-id/354645.1677511842%40sss.pgh.pa.us
---
src/backend/utils/activity/pgstat_io.c | 12 ++++++------
src/backend/utils/adt/pgstatfuncs.c | 8 ++++----
2 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index c478b126fa..c4199d18c8 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -36,16 +36,16 @@ pgstat_bktype_io_stats_valid(PgStat_BktypeIO *backend_io,
{
bool bktype_tracked = pgstat_tracks_io_bktype(bktype);
- for (IOObject io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
{
- for (IOContext io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
{
/*
* Don't bother trying to skip to the next loop iteration if
* pgstat_tracks_io_object() would return false here. We still
* need to validate that each counter is zero anyway.
*/
- for (IOOp io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
+ for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
{
/* No stats, so nothing to validate */
if (backend_io->data[io_object][io_context][io_op] == 0)
@@ -109,11 +109,11 @@ pgstat_flush_io(bool nowait)
else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
return true;
- for (IOObject io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
+ for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
{
- for (IOContext io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
{
- for (IOOp io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
+ for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
bktype_shstats->data[io_object][io_context][io_op] +=
PendingIOStats.data[io_object][io_context][io_op];
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 12eda4ade0..b61a12382b 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1306,7 +1306,7 @@ pg_stat_get_io(PG_FUNCTION_ARGS)
reset_time = TimestampTzGetDatum(backends_io_stats->stat_reset_timestamp);
- for (BackendType bktype = 0; bktype < BACKEND_NUM_TYPES; bktype++)
+ for (int bktype = 0; bktype < BACKEND_NUM_TYPES; bktype++)
{
Datum bktype_desc = CStringGetTextDatum(GetBackendTypeDesc(bktype));
PgStat_BktypeIO *bktype_stats = &backends_io_stats->stats[bktype];
@@ -1325,11 +1325,11 @@ pg_stat_get_io(PG_FUNCTION_ARGS)
if (!pgstat_tracks_io_bktype(bktype))
continue;
- for (IOObject io_obj = 0; io_obj < IOOBJECT_NUM_TYPES; io_obj++)
+ for (int io_obj = 0; io_obj < IOOBJECT_NUM_TYPES; io_obj++)
{
const char *obj_name = pgstat_get_io_object_name(io_obj);
- for (IOContext io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
{
const char *context_name = pgstat_get_io_context_name(io_context);
@@ -1357,7 +1357,7 @@ pg_stat_get_io(PG_FUNCTION_ARGS)
*/
values[IO_COL_CONVERSION] = Int64GetDatum(BLCKSZ);
- for (IOOp io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
+ for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
{
int col_idx = pgstat_get_io_op_index(io_op);
--
2.37.2
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-26 18:52 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 14:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-02-27 15:30 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 19:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-02-27 19:58 ` Tom Lane <[email protected]>
2023-02-27 23:18 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Tom Lane @ 2023-02-27 19:58 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Andres Freund <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
Melanie Plageman <[email protected]> writes:
> On Mon, Feb 27, 2023 at 10:30 AM Tom Lane <[email protected]> wrote:
>> The risk of needing to cast when using the "int" loop variable
>> as an enum is obviously the downside of that approach, but we have
>> not seen any indication that any compilers actually do warn.
>> It's interesting that you did see such a warning ... I wonder which
>> compiler you were using at the time?
> so, pretty much any version of clang I tried with
> -Wsign-conversion produces a warning.
> <source>:35:32: warning: implicit conversion changes signedness: 'int'
> to 'IOOp' (aka 'enum IOOp') [-Wsign-conversion]
Oh, interesting --- so it's not about the implicit conversion to enum
but just about signedness. I bet we could silence that by making the
loop variables be "unsigned int". I doubt it's worth any extra keystrokes
though, because we are not at all clean about sign-conversion warnings.
I tried enabling -Wsign-conversion on Apple's clang 14.0.0 just now,
and counted 13462 such warnings just in the core build :-(. I don't
foresee anybody trying to clean that up.
> I didn't do the casts in the attached patch since they aren't done elsewhere.
Agreed. I'll push this along with the earlier patch if there are
not objections.
regards, tom lane
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-26 18:52 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 14:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-02-27 15:30 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 19:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-02-27 19:58 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
@ 2023-02-27 23:18 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Andres Freund @ 2023-02-27 23:18 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Melanie Plageman <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]
On 2023-02-27 14:58:30 -0500, Tom Lane wrote:
> Agreed. I'll push this along with the earlier patch if there are
> not objections.
None here.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-18 06:10 ` Maciek Sakrejda <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Maciek Sakrejda @ 2023-01-18 06:10 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Andres Freund <[email protected]>; Justin Pryzby <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; vignesh C <[email protected]>
On Tue, Jan 17, 2023 at 9:22 AM Melanie Plageman
<[email protected]> wrote:
> On Mon, Jan 16, 2023 at 4:42 PM Maciek Sakrejda <[email protected]> wrote:
> > I missed a couple of versions, but I think the docs are clearer now.
> > I'm torn on losing some of the detail, but overall I do think it's a
> > good trade-off. Moving some details out to after the table does keep
> > the bulk of the view documentation more readable, and the "inform
> > database tuning" part is great. I really like the idea of a separate
> > Interpreting Statistics section, but for now this works.
> >
> > >+ <literal>vacuum</literal>: I/O operations performed outside of shared
> > >+ buffers while vacuuming and analyzing permanent relations.
> >
> > Why only permanent relations? Are temporary relations treated
> > differently? I imagine if someone has a temp-table-heavy workload that
> > requires regularly vacuuming and analyzing those relations, this point
> > may be confusing without some additional explanation.
>
> Ah, yes. This is a bit confusing. We don't use buffer access strategies
> when operating on temp relations, so vacuuming them is counted in IO
> Context normal. I've added this information to the docs but now that
> definition is a bit long. Perhaps it should be a note? That seems like
> it would draw too much attention to this detail, though...
Thanks for clarifying. I think the updated definition still works:
it's still shorter than the `normal` context definition.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
@ 2023-01-16 21:41 ` Maciek Sakrejda <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Maciek Sakrejda @ 2023-01-16 21:41 UTC (permalink / raw)
To: Melanie Plageman <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Lukas Fittl <[email protected]>; Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers; Thomas Munro <[email protected]>; vignesh C <[email protected]>
On Fri, Jan 13, 2023 at 10:38 AM Melanie Plageman
<[email protected]> wrote:
>
> Attached is v47.
I missed a couple of versions, but I think the docs are clearer now.
I'm torn on losing some of the detail, but overall I do think it's a
good trade-off. Moving some details out to after the table does keep
the bulk of the view documentation more readable, and the "inform
database tuning" part is great. I really like the idea of a separate
Interpreting Statistics section, but for now this works.
>+ <literal>vacuum</literal>: I/O operations performed outside of shared
>+ buffers while vacuuming and analyzing permanent relations.
Why only permanent relations? Are temporary relations treated
differently? I imagine if someone has a temp-table-heavy workload that
requires regularly vacuuming and analyzing those relations, this point
may be confusing without some additional explanation.
Other than that, this looks great.
Thanks,
Maciek
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v29 06/11] Add Incremental View Maintenance support
@ 2023-05-31 09:59 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Yugo Nagata @ 2023-05-31 09:59 UTC (permalink / raw)
In this implementation, AFTER triggers are used to collect
tuplestores containing transition table contents. When multiple tables
are changed, multiple AFTER triggers are invoked, then the final AFTER
trigger performs actual update of the matview. In addition, BEFORE
triggers are also used to handle global information for view
maintenance.
To calculate view deltas, we need both pre-state and post-state of base
tables. Post-update states are available in AFTER trigger, and pre-update
states can be calculated by removing inserted tuples and appending deleted
tuples. Insterted tuples are filtered using the snapshot taken before
table modiication, and deleted tuples are contained in the old transition
table.
Incrementally Maintainable Materialized Views (IMMV) can contain
duplicated tuples.
This patch also allows self-join, simultaneous updates of more than
one base table, and multiple updates of the same base table.
---
src/backend/access/transam/xact.c | 5 +
src/backend/commands/createas.c | 680 +++++++++++++
src/backend/commands/matview.c | 1466 ++++++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 10 +
src/include/commands/createas.h | 4 +
src/include/commands/matview.h | 9 +
6 files changed, 2139 insertions(+), 35 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8daaa535ed..cd280bdffd 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -36,6 +36,7 @@
#include "catalog/pg_enum.h"
#include "catalog/storage.h"
#include "commands/async.h"
+#include "commands/matview.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "common/pg_prng.h"
@@ -2803,6 +2804,7 @@ AbortTransaction(void)
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
AtAbort_Twophase();
+ AtAbort_IVM();
/*
* Advertise the fact that we aborted in pg_xact (assuming that we got as
@@ -5080,6 +5082,9 @@ AbortSubTransaction(void)
pgstat_progress_end_command();
UnlockBuffers();
+ /* Clean up hash entries for incremental view maintenance */
+ AtAbort_IVM();
+
/* Reset WAL record construction state */
XLogResetInsertion();
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index e91920ca14..415f110516 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -32,15 +32,26 @@
#include "access/xact.h"
#include "access/xlog.h"
#include "catalog/namespace.h"
+#include "catalog/index.h"
+#include "catalog/pg_constraint.h"
+#include "catalog/pg_inherits.h"
+#include "catalog/pg_trigger.h"
#include "catalog/toasting.h"
#include "commands/createas.h"
+#include "commands/defrem.h"
#include "commands/matview.h"
#include "commands/prepare.h"
#include "commands/tablecmds.h"
+#include "commands/tablespace.h"
+#include "commands/trigger.h"
#include "commands/view.h"
#include "miscadmin.h"
+#include "optimizer/optimizer.h"
+#include "optimizer/prep.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
#include "parser/parse_clause.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
@@ -73,6 +84,12 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTriggersOnBaseTablesRecurse(Query *qry, Node *node, Oid matviewOid,
+ Relids *relids, bool ex_lock);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type, int16 timing, bool ex_lock);
+static void check_ivm_restriction(Node *node);
+static bool check_ivm_restriction_walker(Node *node, void *context);
+static Bitmapset *get_primary_key_attnos_from_query(Query *query, List **constraintList);
/*
* create_ctas_internal
@@ -282,6 +299,18 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
save_nestlevel = NewGUCNestLevel();
}
+ if (is_matview && into->ivm)
+ {
+ /* check if the query is supported in IMMV definition */
+ if (contain_mutable_functions((Node *) query))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("mutable function is not supported on incrementally maintainable materialized view"),
+ errhint("functions must be marked IMMUTABLE")));
+
+ check_ivm_restriction((Node *) query);
+ }
+
if (into->skipData)
{
/*
@@ -358,6 +387,27 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ if (into->ivm)
+ {
+ Oid matviewOid = address.objectId;
+ Relation matviewRel = table_open(matviewOid, NoLock);
+
+ /*
+ * Mark relisivm field, if it's a matview and into->ivm is true.
+ */
+ SetMatViewIVMState(matviewRel, true);
+
+ if (!into->skipData)
+ {
+ /* Create an index on incremental maintainable materialized view, if possible */
+ CreateIndexOnIMMV((Query *) into->viewQuery, matviewRel);
+
+ /* Create triggers on incremental maintainable materialized view */
+ CreateIvmTriggersOnBaseTables((Query *) into->viewQuery, matviewOid);
+ }
+ table_close(matviewRel, NoLock);
+ }
}
return address;
@@ -635,3 +685,633 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+/*
+ * CreateIvmTriggersOnBaseTables -- create IVM triggers on all base tables
+ */
+void
+CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid)
+{
+ Relids relids = NULL;
+ bool ex_lock = false;
+ RangeTblEntry *rte;
+
+ /* Immediately return if we don't have any base tables. */
+ if (list_length(qry->rtable) < 1)
+ return;
+
+ /*
+ * If the view has more than one base tables, we need an exclusive lock
+ * on the view so that the view would be maintained serially to avoid
+ * the inconsistency that occurs when two base tables are modified in
+ * concurrent transactions. However, if the view has only one table,
+ * we can use a weaker lock.
+ *
+ * The type of lock should be determined here, because if we check the
+ * view definition at maintenance time, we need to acquire a weaker lock,
+ * and upgrading the lock level after this increases probability of
+ * deadlock.
+ */
+
+ rte = list_nth(qry->rtable, 0);
+ if (list_length(qry->rtable) > 1 || rte->rtekind != RTE_RELATION)
+ ex_lock = true;
+
+ CreateIvmTriggersOnBaseTablesRecurse(qry, (Node *)qry, matviewOid, &relids, ex_lock);
+
+ bms_free(relids);
+}
+
+static void
+CreateIvmTriggersOnBaseTablesRecurse(Query *qry, Node *node, Oid matviewOid,
+ Relids *relids, bool ex_lock)
+{
+ if (node == NULL)
+ return;
+
+ /* This can recurse, so check for excessive recursion */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_Query:
+ {
+ Query *query = (Query *) node;
+
+ CreateIvmTriggersOnBaseTablesRecurse(qry, (Node *)query->jointree, matviewOid, relids, ex_lock);
+ }
+ break;
+
+ case T_RangeTblRef:
+ {
+ int rti = ((RangeTblRef *) node)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION && !bms_is_member(rte->relid, *relids))
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_BEFORE, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_BEFORE, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_BEFORE, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_TRUNCATE, TRIGGER_TYPE_BEFORE, true);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_AFTER, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_AFTER, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_AFTER, ex_lock);
+ CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_TRUNCATE, TRIGGER_TYPE_AFTER, true);
+
+ *relids = bms_add_member(*relids, rte->relid);
+ }
+ }
+ break;
+
+ case T_FromExpr:
+ {
+ FromExpr *f = (FromExpr *) node;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTablesRecurse(qry, lfirst(l), matviewOid, relids, ex_lock);
+ }
+ break;
+
+ case T_JoinExpr:
+ {
+ JoinExpr *j = (JoinExpr *) node;
+
+ CreateIvmTriggersOnBaseTablesRecurse(qry, j->larg, matviewOid, relids, ex_lock);
+ CreateIvmTriggersOnBaseTablesRecurse(qry, j->rarg, matviewOid, relids, ex_lock);
+ }
+ break;
+
+ default:
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node));
+ }
+}
+
+/*
+ * CreateIvmTrigger -- create IVM trigger on a base table
+ */
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, int16 type, int16 timing, bool ex_lock)
+{
+ ObjectAddress refaddr;
+ ObjectAddress address;
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+
+ Assert(timing == TRIGGER_TYPE_BEFORE || timing == TRIGGER_TYPE_AFTER);
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+
+ ivm_trigger->timing = timing;
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_ins_before" : "IVM_trigger_ins_after");
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_del_before" : "IVM_trigger_del_after");
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_upd_before" : "IVM_trigger_upd_after");
+ break;
+ case TRIGGER_TYPE_TRUNCATE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_truncate_before" : "IVM_trigger_truncate_after");
+ break;
+ default:
+ elog(ERROR, "unsupported trigger type");
+ }
+
+ if (timing == TRIGGER_TYPE_AFTER)
+ {
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "__ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "__ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ }
+
+ /*
+ * XXX: When using DELETE or UPDATE, we must use exclusive lock for now
+ * because apply_old_delta(_with_count) uses ctid to identify the tuple
+ * to be deleted/deleted, but doesn't work in concurrent situations.
+ *
+ * If the view doesn't have aggregate, distinct, or tuple duplicate,
+ * then it would work even in concurrent situations. However, we don't have
+ * any way to guarantee the view has a unique key before opening the IMMV
+ * at the maintenance time because users may drop the unique index.
+ */
+
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ ex_lock = true;
+
+ ivm_trigger->funcname =
+ (timing == TRIGGER_TYPE_BEFORE ? SystemFuncName("IVM_immediate_before") : SystemFuncName("IVM_immediate_maintenance"));
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make2(
+ makeString(DatumGetPointer(DirectFunctionCall1(oidout, ObjectIdGetDatum(viewOid)))),
+ makeString(DatumGetPointer(DirectFunctionCall1(boolout, BoolGetDatum(ex_lock))))
+ );
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
+
+/*
+ * check_ivm_restriction --- look for specify nodes in the query tree
+ */
+static void
+check_ivm_restriction(Node *node)
+{
+ check_ivm_restriction_walker(node, NULL);
+}
+
+static bool
+check_ivm_restriction_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+
+ /*
+ * We currently don't support Sub-Query.
+ */
+ if (IsA(node, SubPlan) || IsA(node, SubLink))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("subquery is not supported on incrementally maintainable materialized view")));
+
+ /* This can recurse, so check for excessive recursion */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_Query:
+ {
+ Query *qry = (Query *)node;
+ ListCell *lc;
+ List *vars;
+
+ /* if contained CTE, return error */
+ if (qry->cteList != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("CTE is not supported on incrementally maintainable materialized view")));
+ if (qry->havingQual != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg(" HAVING clause is not supported on incrementally maintainable materialized view")));
+ if (qry->sortClause != NIL) /* There is a possibility that we don't need to return an error */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("ORDER BY clause is not supported on incrementally maintainable materialized view")));
+ if (qry->limitOffset != NULL || qry->limitCount != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
+ if (qry->distinctClause)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
+ if (qry->hasDistinctOn)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("DISTINCT ON is not supported on incrementally maintainable materialized view")));
+ if (qry->hasWindowFuncs)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("window functions are not supported on incrementally maintainable materialized view")));
+ if (qry->groupingSets != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("GROUPING SETS, ROLLUP, or CUBE clauses is not supported on incrementally maintainable materialized view")));
+ if (qry->setOperations != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("UNION/INTERSECT/EXCEPT statements are not supported on incrementally maintainable materialized view")));
+ if (list_length(qry->targetList) == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("empty target list is not supported on incrementally maintainable materialized view")));
+ if (qry->rowMarks != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("FOR UPDATE/SHARE clause is not supported on incrementally maintainable materialized view")));
+
+ /* system column restrictions */
+ vars = pull_vars_of_level((Node *) qry, 0);
+ foreach(lc, vars)
+ {
+ if (IsA(lfirst(lc), Var))
+ {
+ Var *var = (Var *) lfirst(lc);
+ /* if system column, return error */
+ if (var->varattno < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("system column is not supported on incrementally maintainable materialized view")));
+ }
+ }
+
+ /* restrictions for rtable */
+ foreach(lc, qry->rtable)
+ {
+ RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc);
+
+ if (rte->subquery)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("subquery is not supported on incrementally maintainable materialized view")));
+
+ if (rte->tablesample != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("TABLESAMPLE clause is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("partitioned table is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_RELATION && has_superclass(rte->relid))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("partitions is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_RELATION && find_inheritance_children(rte->relid, NoLock) != NIL)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("inheritance parent is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_FOREIGN_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("foreign table is not supported on incrementally maintainable materialized view")));
+
+ if (rte->relkind == RELKIND_VIEW ||
+ rte->relkind == RELKIND_MATVIEW)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("VIEW or MATERIALIZED VIEW is not supported on incrementally maintainable materialized view")));
+
+ if (rte->rtekind == RTE_VALUES)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("VALUES is not supported on incrementally maintainable materialized view")));
+
+ }
+
+ query_tree_walker(qry, check_ivm_restriction_walker, NULL, QTW_IGNORE_RANGE_TABLE);
+
+ break;
+ }
+ case T_TargetEntry:
+ {
+ TargetEntry *tle = (TargetEntry *)node;
+ if (isIvmName(tle->resname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column name %s is not supported on incrementally maintainable materialized view", tle->resname)));
+
+ expression_tree_walker(node, check_ivm_restriction_walker, NULL);
+ break;
+ }
+ case T_JoinExpr:
+ {
+ JoinExpr *joinexpr = (JoinExpr *)node;
+
+ if (joinexpr->jointype > JOIN_INNER)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("OUTER JOIN is not supported on incrementally maintainable materialized view")));
+
+ expression_tree_walker(node, check_ivm_restriction_walker, NULL);
+ }
+ break;
+ case T_Aggref:
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("aggregate function is not supported on incrementally maintainable materialized view")));
+ break;
+ default:
+ expression_tree_walker(node, check_ivm_restriction_walker, (void *) context);
+ break;
+ }
+ return false;
+}
+
+/*
+ * CreateIndexOnIMMV
+ *
+ * Create a unique index on incremental maintainable materialized view.
+ * If the view definition query has a GROUP BY clause, the index is created
+ * on the columns of GROUP BY expressions. Otherwise, if the view contains
+ * all primary key attritubes of its base tables in the target list, the index
+ * is created on these attritubes. In other cases, no index is created.
+ */
+void
+CreateIndexOnIMMV(Query *query, Relation matviewRel)
+{
+ ListCell *lc;
+ IndexStmt *index;
+ ObjectAddress address;
+ List *constraintList = NIL;
+ char idxname[NAMEDATALEN];
+ List *indexoidlist = RelationGetIndexList(matviewRel);
+ ListCell *indexoidscan;
+ Bitmapset *key_attnos;
+
+ snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
+
+ index = makeNode(IndexStmt);
+
+ index->unique = true;
+ index->primary = false;
+ index->isconstraint = false;
+ index->deferrable = false;
+ index->initdeferred = false;
+ index->idxname = idxname;
+ index->relation =
+ makeRangeVar(get_namespace_name(RelationGetNamespace(matviewRel)),
+ pstrdup(RelationGetRelationName(matviewRel)),
+ -1);
+ index->accessMethod = DEFAULT_INDEX_TYPE;
+ index->options = NIL;
+ index->tableSpace = get_tablespace_name(matviewRel->rd_rel->reltablespace);
+ index->whereClause = NULL;
+ index->indexParams = NIL;
+ index->indexIncludingParams = NIL;
+ index->excludeOpNames = NIL;
+ index->idxcomment = NULL;
+ index->indexOid = InvalidOid;
+ index->oldNumber = InvalidRelFileNumber;
+ index->oldCreateSubid = InvalidSubTransactionId;
+ index->oldFirstRelfilelocatorSubid = InvalidSubTransactionId;
+ index->transformed = true;
+ index->concurrent = false;
+ index->if_not_exists = false;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns. "),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
+
+ /* If we have a compatible index, we don't need to create another. */
+ foreach(indexoidscan, indexoidlist)
+ {
+ Oid indexoid = lfirst_oid(indexoidscan);
+ Relation indexRel;
+ bool hasCompatibleIndex = false;
+
+ indexRel = index_open(indexoid, AccessShareLock);
+
+ if (CheckIndexCompatible(indexRel->rd_id,
+ index->accessMethod,
+ index->indexParams,
+ index->excludeOpNames))
+ hasCompatibleIndex = true;
+
+ index_close(indexRel, AccessShareLock);
+
+ if (hasCompatibleIndex)
+ return;
+ }
+
+ address = DefineIndex(RelationGetRelid(matviewRel),
+ index,
+ InvalidOid,
+ InvalidOid,
+ InvalidOid,
+ -1,
+ false, true, false, false, true);
+
+ ereport(NOTICE,
+ (errmsg("created index \"%s\" on materialized view \"%s\"",
+ idxname, RelationGetRelationName(matviewRel))));
+
+ /*
+ * Make dependencies so that the index is dropped if any base tables's
+ * primary key is dropped.
+ */
+ foreach(lc, constraintList)
+ {
+ Oid constraintOid = lfirst_oid(lc);
+ ObjectAddress refaddr;
+
+ refaddr.classId = ConstraintRelationId;
+ refaddr.objectId = constraintOid;
+ refaddr.objectSubId = 0;
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_NORMAL);
+ }
+}
+
+
+/*
+ * get_primary_key_attnos_from_query
+ *
+ * Identify the columns in base tables' primary keys in the target list.
+ *
+ * Returns a Bitmapset of the column attnos of the primary key's columns of
+ * tables that used in the query. The attnos are offset by
+ * FirstLowInvalidHeapAttributeNumber as same as get_primary_key_attnos.
+ *
+ * If any table has no primary key or any primary key's columns is not in
+ * the target list, return NULL. We also return NULL if any pkey constraint
+ * is deferrable.
+ *
+ * constraintList is set to a list of the OIDs of the pkey constraints.
+ */
+static Bitmapset *
+get_primary_key_attnos_from_query(Query *query, List **constraintList)
+{
+ List *key_attnos_list = NIL;
+ ListCell *lc;
+ int i;
+ Bitmapset *keys = NULL;
+ Relids rels_in_from;
+
+ /*
+ * Collect primary key attributes from all tables used in query. The key attributes
+ * sets for each table are stored in key_attnos_list in order by RTE index.
+ */
+ foreach(lc, query->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ Bitmapset *key_attnos;
+ bool has_pkey = true;
+
+ /* for tables, call get_primary_key_attnos */
+ if (r->rtekind == RTE_RELATION)
+ {
+ Oid constraintOid;
+ key_attnos = get_primary_key_attnos(r->relid, false, &constraintOid);
+ *constraintList = lappend_oid(*constraintList, constraintOid);
+ has_pkey = (key_attnos != NULL);
+ }
+ /* for other RTEs, store NULL into key_attnos_list */
+ else
+ key_attnos = NULL;
+
+ /*
+ * If any table or subquery has no primary key or its pkey constraint is deferrable,
+ * we cannot get key attributes for this query, so return NULL.
+ */
+ if (!has_pkey)
+ return NULL;
+
+ key_attnos_list = lappend(key_attnos_list, key_attnos);
+ }
+
+ /* Collect key attributes appearing in the target list */
+ i = 1;
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) flatten_join_alias_vars(NULL, query, lfirst(lc));
+
+ if (IsA(tle->expr, Var))
+ {
+ Var *var = (Var*) tle->expr;
+ Bitmapset *key_attnos = list_nth(key_attnos_list, var->varno - 1);
+
+ /* check if this attribute is from a base table's primary key */
+ if (bms_is_member(var->varattno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ /*
+ * Remove found key attributes from key_attnos_list, and add this
+ * to the result list.
+ */
+ key_attnos = bms_del_member(key_attnos, var->varattno - FirstLowInvalidHeapAttributeNumber);
+ if (bms_is_empty(key_attnos))
+ {
+ key_attnos_list = list_delete_nth_cell(key_attnos_list, var->varno - 1);
+ key_attnos_list = list_insert_nth(key_attnos_list, var->varno - 1, NULL);
+ }
+ keys = bms_add_member(keys, i - FirstLowInvalidHeapAttributeNumber);
+ }
+ }
+ i++;
+ }
+
+ /* Collect RTE indexes of relations appearing in the FROM clause */
+ rels_in_from = get_relids_in_jointree((Node *) query->jointree, false, false);
+
+ /*
+ * Check if all key attributes of relations in FROM are appearing in the target
+ * list. If an attribute remains in key_attnos_list in spite of the table is used
+ * in FROM clause, the target is missing this key attribute, so we return NULL.
+ */
+ i = 1;
+ foreach(lc, key_attnos_list)
+ {
+ Bitmapset *bms = (Bitmapset *)lfirst(lc);
+ if (!bms_is_empty(bms) && bms_is_member(i, rels_in_from))
+ return NULL;
+ i++;
+ }
+
+ return keys;
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index ac2e74fa3f..39305f3c49 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -25,26 +25,37 @@
#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_am.h"
+#include "catalog/pg_depend.h"
+#include "catalog/pg_trigger.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
#include "commands/cluster.h"
#include "commands/matview.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
+#include "commands/createas.h"
#include "executor/executor.h"
#include "executor/spi.h"
+#include "executor/tstoreReceiver.h"
#include "miscadmin.h"
+#include "nodes/makefuncs.h"
+#include "parser/analyze.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "parser/parse_relation.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
+#include "rewrite/rowsecurity.h"
#include "storage/lmgr.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/typcache.h"
typedef struct
@@ -58,6 +69,52 @@ typedef struct
BulkInsertState bistate; /* bulk insert state */
} DR_transientrel;
+#define MV_INIT_QUERYHASHSIZE 16
+
+/*
+ * MV_TriggerHashEntry
+ *
+ * Hash entry for base tables on which IVM trigger is invoked
+ */
+typedef struct MV_TriggerHashEntry
+{
+ Oid matview_id; /* OID of the materialized view */
+ int before_trig_count; /* count of before triggers invoked */
+ int after_trig_count; /* count of after triggers invoked */
+
+ Snapshot snapshot; /* Snapshot just before table change */
+
+ List *tables; /* List of MV_TriggerTable */
+ bool has_old; /* tuples are deleted from any table? */
+ bool has_new; /* tuples are inserted into any table? */
+} MV_TriggerHashEntry;
+
+/*
+ * MV_TriggerTable
+ *
+ * IVM related data for tables on which the trigger is invoked.
+ */
+typedef struct MV_TriggerTable
+{
+ Oid table_id; /* OID of the modified table */
+ List *old_tuplestores; /* tuplestores for deleted tuples */
+ List *new_tuplestores; /* tuplestores for inserted tuples */
+
+ List *rte_indexes; /* List of RTE index of the modified table */
+ RangeTblEntry *original_rte; /* the original RTE saved before rewriting query */
+
+ Relation rel; /* relation of the modified table */
+ TupleTableSlot *slot; /* for checking visibility in the pre-state table */
+} MV_TriggerTable;
+
+static HTAB *mv_trigger_info = NULL;
+
+static bool in_delta_calculation = false;
+
+/* ENR name for materialized view delta */
+#define NEW_DELTA_ENRNAME "new_delta"
+#define OLD_DELTA_ENRNAME "old_delta"
+
static int matview_maintenance_depth = 0;
static void transientrel_startup(DestReceiver *self, int operation, TupleDesc typeinfo);
@@ -65,7 +122,9 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
- const char *queryString);
+ QueryEnvironment *queryEnv,
+ TupleDesc *resultTupleDesc,
+ const char *queryString);
static char *make_temptable_name_n(char *tempname, int n);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
int save_sec_context);
@@ -73,6 +132,37 @@ static void refresh_by_heap_swap(Oid matviewOid, Oid OIDNewHeap, char relpersist
static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static Query *get_matview_query(Relation matviewRel);
+
+static Query *rewrite_query_for_preupdate_state(Query *query, List *tables,
+ ParseState *pstate, Oid matviewid);
+static void register_delta_ENRs(ParseState *pstate, Query *query, List *tables);
+static char *make_delta_enr_name(const char *prefix, Oid relid, int count);
+static RangeTblEntry *get_prestate_rte(RangeTblEntry *rte, MV_TriggerTable *table,
+ QueryEnvironment *queryEnv, Oid matviewid);
+static RangeTblEntry *replace_rte_with_delta(RangeTblEntry *rte, MV_TriggerTable *table, bool is_new,
+ QueryEnvironment *queryEnv);
+static Query *rewrite_query_for_counting(Query *query, ParseState *pstate);
+
+static void calc_delta(MV_TriggerTable *table, int rte_index, Query *query,
+ DestReceiver *dest_old, DestReceiver *dest_new,
+ TupleDesc *tupdesc_old, TupleDesc *tupdesc_new,
+ QueryEnvironment *queryEnv);
+static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte_index);
+
+static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
+ TupleDesc tupdesc_old, TupleDesc tupdesc_new,
+ Query *query);
+static void apply_old_delta(const char *matviewname, const char *deltaname_old,
+ List *keys);
+static void apply_new_delta(const char *matviewname, const char *deltaname_new,
+ StringInfo target_list);
+static char *get_matching_condition_string(List *keys);
+static void generate_equal(StringInfo querybuf, Oid opttype,
+ const char *leftop, const char *rightop);
+
+static void mv_InitHashTables(void);
+static void clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry, bool is_abort);
/*
* SetMatViewPopulatedState
@@ -114,6 +204,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
CommandCounterIncrement();
}
+/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
@@ -140,8 +270,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
{
Oid matviewOid;
Relation matviewRel;
- RewriteRule *rule;
- List *actions;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -155,6 +283,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
int save_sec_context;
int save_nestlevel;
ObjectAddress address;
+ bool oldPopulated;
/* Determine strength of lock needed. */
concurrent = stmt->concurrent;
@@ -178,6 +307,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
SetUserIdAndSecContext(relowner,
save_sec_context | SECURITY_RESTRICTED_OPERATION);
save_nestlevel = NewGUCNestLevel();
+ oldPopulated = RelationIsPopulated(matviewRel);
/* Make sure it is a materialized view. */
if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW)
@@ -199,32 +329,9 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
errmsg("%s and %s options cannot be used together",
"CONCURRENTLY", "WITH NO DATA")));
- /*
- * Check that everything is correct for a refresh. Problems at this point
- * are internal errors, so elog is sufficient.
- */
- if (matviewRel->rd_rel->relhasrules == false ||
- matviewRel->rd_rules->numLocks < 1)
- elog(ERROR,
- "materialized view \"%s\" is missing rewrite information",
- RelationGetRelationName(matviewRel));
-
- if (matviewRel->rd_rules->numLocks > 1)
- elog(ERROR,
- "materialized view \"%s\" has too many rules",
- RelationGetRelationName(matviewRel));
- rule = matviewRel->rd_rules->rules[0];
- if (rule->event != CMD_SELECT || !(rule->isInstead))
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
- RelationGetRelationName(matviewRel));
+ dataQuery = get_matview_query(matviewRel);
- actions = rule->actions;
- if (list_length(actions) != 1)
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a single action",
- RelationGetRelationName(matviewRel));
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -259,12 +366,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
errhint("Create a unique index with no WHERE clause on one or more columns of the materialized view.")));
}
- /*
- * The stored query was rewritten at the time of the MV definition, but
- * has not been scribbled on by the planner.
- */
- dataQuery = linitial_node(Query, actions);
-
/*
* Check for active uses of the relation in the current transaction, such
* as open scans.
@@ -292,6 +393,74 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
relpersistence = matviewRel->rd_rel->relpersistence;
}
+ /* delete IMMV triggers. */
+ if (RelationIsIVM(matviewRel) && stmt->skipData )
+ {
+ Relation tgRel;
+ Relation depRel;
+ ScanKeyData key;
+ SysScanDesc scan;
+ HeapTuple tup;
+ ObjectAddresses *immv_triggers;
+
+ immv_triggers = new_object_addresses();
+
+ tgRel = table_open(TriggerRelationId, RowExclusiveLock);
+ depRel = table_open(DependRelationId, RowExclusiveLock);
+
+ /* search triggers that depends on IMMV. */
+ ScanKeyInit(&key,
+ Anum_pg_depend_refobjid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(matviewOid));
+ scan = systable_beginscan(depRel, DependReferenceIndexId, true,
+ NULL, 1, &key);
+ while ((tup = systable_getnext(scan)) != NULL)
+ {
+ ObjectAddress obj;
+ Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (foundDep->classid == TriggerRelationId)
+ {
+ HeapTuple tgtup;
+ ScanKeyData tgkey[1];
+ SysScanDesc tgscan;
+ Form_pg_trigger tgform;
+
+ /* Find the trigger name. */
+ ScanKeyInit(&tgkey[0],
+ Anum_pg_trigger_oid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(foundDep->objid));
+
+ tgscan = systable_beginscan(tgRel, TriggerOidIndexId, true,
+ NULL, 1, tgkey);
+ tgtup = systable_getnext(tgscan);
+ if (!HeapTupleIsValid(tgtup))
+ elog(ERROR, "could not find tuple for immv trigger %u", foundDep->objid);
+
+ tgform = (Form_pg_trigger) GETSTRUCT(tgtup);
+
+ /* If trigger is created by IMMV, delete it. */
+ if (strncmp(NameStr(tgform->tgname), "IVM_trigger_", 12) == 0)
+ {
+ obj.classId = foundDep->classid;
+ obj.objectId = foundDep->objid;
+ obj.objectSubId = foundDep->refobjsubid;
+ add_exact_object_address(&obj, immv_triggers);
+ }
+ systable_endscan(tgscan);
+ }
+ }
+ systable_endscan(scan);
+
+ performMultipleDeletions(immv_triggers, DROP_RESTRICT, PERFORM_DELETION_INTERNAL);
+
+ table_close(depRel, RowExclusiveLock);
+ table_close(tgRel, RowExclusiveLock);
+ free_object_addresses(immv_triggers);
+ }
+
/*
* Create the transient table that will receive the regenerated data. Lock
* it against access by any other process until commit (by which time it
@@ -305,7 +474,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
/* Generate the data, if wanted. */
if (!stmt->skipData)
- processed = refresh_matview_datafill(dest, dataQuery, queryString);
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, NULL, queryString);
/* Make the matview match the newly generated data. */
if (concurrent)
@@ -340,6 +509,12 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
pgstat_count_heap_insert(matviewRel, processed);
}
+ if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
+ {
+ CreateIndexOnIMMV(dataQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ }
+
table_close(matviewRel, NoLock);
/* Roll back any GUC changes */
@@ -374,6 +549,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
+ TupleDesc *resultTupleDesc,
const char *queryString)
{
List *rewritten;
@@ -410,7 +587,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -420,6 +597,9 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
processed = queryDesc->estate->es_processed;
+ if (resultTupleDesc)
+ *resultTupleDesc = CreateTupleDescCopy(queryDesc->tupDesc);
+
/* and clean up */
ExecutorFinish(queryDesc);
ExecutorEnd(queryDesc);
@@ -931,3 +1111,1219 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * get_matview_query - get the Query from a matview's _RETURN rule.
+ */
+static Query *
+get_matview_query(Relation matviewRel)
+{
+ RewriteRule *rule;
+ List * actions;
+
+ /*
+ * Check that everything is correct for a refresh. Problems at this point
+ * are internal errors, so elog is sufficient.
+ */
+ if (matviewRel->rd_rel->relhasrules == false ||
+ matviewRel->rd_rules->numLocks < 1)
+ elog(ERROR,
+ "materialized view \"%s\" is missing rewrite information",
+ RelationGetRelationName(matviewRel));
+
+ if (matviewRel->rd_rules->numLocks > 1)
+ elog(ERROR,
+ "materialized view \"%s\" has too many rules",
+ RelationGetRelationName(matviewRel));
+
+ rule = matviewRel->rd_rules->rules[0];
+ if (rule->event != CMD_SELECT || !(rule->isInstead))
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
+ RelationGetRelationName(matviewRel));
+
+ actions = rule->actions;
+ if (list_length(actions) != 1)
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a single action",
+ RelationGetRelationName(matviewRel));
+
+ /*
+ * The stored query was rewritten at the time of the MV definition, but
+ * has not been scribbled on by the planner.
+ */
+ return linitial_node(Query, actions);
+}
+
+
+/* ----------------------------------------------------
+ * Incremental View Maintenance routines
+ * ---------------------------------------------------
+ */
+
+/*
+ * IVM_immediate_before
+ *
+ * IVM trigger function invoked before base table is modified. If this is
+ * invoked firstly in the same statement, we save the transaction id and the
+ * command id at that time.
+ */
+Datum
+IVM_immediate_before(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ char *matviewOid_text = trigdata->tg_trigger->tgargs[0];
+ char *ex_lock_text = trigdata->tg_trigger->tgargs[1];
+ Oid matviewOid;
+ MV_TriggerHashEntry *entry;
+ bool found;
+ bool ex_lock;
+
+ matviewOid = DatumGetObjectId(DirectFunctionCall1(oidin, CStringGetDatum(matviewOid_text)));
+ ex_lock = DatumGetBool(DirectFunctionCall1(boolin, CStringGetDatum(ex_lock_text)));
+
+ /* If the view has more than one tables, we have to use an exclusive lock. */
+ if (ex_lock)
+ {
+ /*
+ * Wait for concurrent transactions which update this materialized view at
+ * READ COMMITED. This is needed to see changes committed in other
+ * transactions. No wait and raise an error at REPEATABLE READ or
+ * SERIALIZABLE to prevent update anomalies of matviews.
+ * XXX: dead-lock is possible here.
+ */
+ if (!IsolationUsesXactSnapshot())
+ LockRelationOid(matviewOid, ExclusiveLock);
+ else if (!ConditionalLockRelationOid(matviewOid, ExclusiveLock))
+ {
+ /* try to throw error by name; relation could be deleted... */
+ char *relname = get_rel_name(matviewOid);
+
+ if (!relname)
+ ereport(ERROR,
+ (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
+ errmsg("could not obtain lock on materialized view during incremental maintenance")));
+
+ ereport(ERROR,
+ (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
+ errmsg("could not obtain lock on materialized view \"%s\" during incremental maintenance",
+ relname)));
+ }
+ }
+ else
+ LockRelationOid(matviewOid, RowExclusiveLock);
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_trigger_info)
+ mv_InitHashTables();
+
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_ENTER, &found);
+
+ /* On the first BEFORE to update the view, initialize trigger data */
+ if (!found)
+ {
+ /*
+ * Get a snapshot just before the table was modified for checking
+ * tuple visibility in the pre-update state of the table.
+ */
+ Snapshot snapshot = GetActiveSnapshot();
+
+ entry->matview_id = matviewOid;
+ entry->before_trig_count = 0;
+ entry->after_trig_count = 0;
+ entry->snapshot = RegisterSnapshot(snapshot);
+ entry->tables = NIL;
+ entry->has_old = false;
+ entry->has_new = false;
+ }
+
+ entry->before_trig_count++;
+
+ return PointerGetDatum(NULL);
+}
+
+/*
+ * IVM_immediate_maintenance
+ *
+ * IVM trigger function invoked after base table is modified.
+ * For each table, tuplestores of transition tables are collected.
+ * and after the last modification
+ */
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query;
+ Query *rewritten = NULL;
+ char *matviewOid_text = trigdata->tg_trigger->tgargs[0];
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+
+ Oid relowner;
+ Tuplestorestate *old_tuplestore = NULL;
+ Tuplestorestate *new_tuplestore = NULL;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ MV_TriggerHashEntry *entry;
+ MV_TriggerTable *table;
+ bool found;
+
+ ParseState *pstate;
+ QueryEnvironment *queryEnv = create_queryEnv();
+ MemoryContext oldcxt;
+ ListCell *lc;
+ int i;
+
+
+ /* Create a ParseState for rewriting the view definition query */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ matviewOid = DatumGetObjectId(DirectFunctionCall1(oidin, CStringGetDatum(matviewOid_text)));
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_trigger_info)
+ mv_InitHashTables();
+
+ /* get the entry for this materialized view */
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_FIND, &found);
+ Assert (found && entry != NULL);
+ entry->after_trig_count++;
+
+ /* search the entry for the modified table and create new entry if not found */
+ found = false;
+ foreach(lc, entry->tables)
+ {
+ table = (MV_TriggerTable *) lfirst(lc);
+ if (table->table_id == relid)
+ {
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+
+ table = (MV_TriggerTable *) palloc0(sizeof(MV_TriggerTable));
+ table->table_id = relid;
+ table->old_tuplestores = NIL;
+ table->new_tuplestores = NIL;
+ table->rte_indexes = NIL;
+ table->slot = MakeSingleTupleTableSlot(RelationGetDescr(rel), table_slot_callbacks(rel));
+ table->rel = table_open(RelationGetRelid(rel), NoLock);
+ entry->tables = lappend(entry->tables, table);
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ /* Save the transition tables and make a request to not free immediately */
+ if (trigdata->tg_oldtable)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+ table->old_tuplestores = lappend(table->old_tuplestores, trigdata->tg_oldtable);
+ entry->has_old = true;
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (trigdata->tg_newtable)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+ table->new_tuplestores = lappend(table->new_tuplestores, trigdata->tg_newtable);
+ entry->has_new = true;
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (entry->has_new || entry->has_old)
+ {
+ CmdType cmd;
+
+ if (TRIGGER_FIRED_BY_INSERT(trigdata->tg_event))
+ cmd = CMD_INSERT;
+ else if (TRIGGER_FIRED_BY_DELETE(trigdata->tg_event))
+ cmd = CMD_DELETE;
+ else if (TRIGGER_FIRED_BY_UPDATE(trigdata->tg_event))
+ cmd = CMD_UPDATE;
+ else
+ elog(ERROR,"unsupported trigger type");
+
+ /* Prolong lifespan of transition tables to the end of the last AFTER trigger */
+ SetTransitionTablePreserved(relid, cmd);
+ }
+
+
+ /* If this is not the last AFTER trigger call, immediately exit. */
+ Assert (entry->before_trig_count >= entry->after_trig_count);
+ if (entry->before_trig_count != entry->after_trig_count)
+ return PointerGetDatum(NULL);
+
+ /*
+ * If this is the last AFTER trigger call, continue and update the view.
+ */
+
+ /*
+ * Advance command counter to make the updated base table row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+
+ matviewRel = table_open(matviewOid, NoLock);
+
+ /* Make sure it is a materialized view. */
+ Assert(matviewRel->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Get and push the latast snapshot to see any changes which is committed
+ * during waiting in other transactions at READ COMMITTED level.
+ */
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ CheckTableNotInUse(matviewRel, "refresh a materialized view incrementally");
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * We will switch modes when we are about to execute user code.
+ */
+ relowner = matviewRel->rd_rel->relowner;
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+ save_nestlevel = NewGUCNestLevel();
+
+ /* get view query*/
+ query = get_matview_query(matviewRel);
+
+ /*
+ * When a base table is truncated, the view content will be empty if the
+ * view definition query does not contain an aggregate without a GROUP clause.
+ * Therefore, such views can be truncated.
+ */
+ if (TRIGGER_FIRED_BY_TRUNCATE(trigdata->tg_event))
+ {
+ ExecuteTruncateGuts(list_make1(matviewRel), list_make1_oid(matviewOid),
+ NIL, DROP_RESTRICT, false, false);
+
+ /* Clean up hash entry and delete tuplestores */
+ clean_up_IVM_hash_entry(entry, false);
+
+ /* Pop the original snapshot. */
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+ }
+
+ /*
+ * rewrite query for calculating deltas
+ */
+
+ rewritten = copyObject(query);
+
+ /* Replace resnames in a target list with materialized view's attnames */
+ i = 0;
+ foreach (lc, rewritten->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+ char *resname = NameStr(attr->attname);
+
+ tle->resname = pstrdup(resname);
+ i++;
+ }
+
+ /* Set all tables in the query to pre-update state */
+ rewritten = rewrite_query_for_preupdate_state(rewritten, entry->tables,
+ pstate, matviewOid);
+ /* Rewrite for counting duplicated tuples */
+ rewritten = rewrite_query_for_counting(rewritten, pstate);
+
+ /* Create tuplestores to store view deltas */
+ if (entry->has_old)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+
+ old_tuplestore = tuplestore_begin_heap(false, false, work_mem);
+ dest_old = CreateDestReceiver(DestTuplestore);
+ SetTuplestoreDestReceiverParams(dest_old,
+ old_tuplestore,
+ TopTransactionContext,
+ false,
+ NULL,
+ NULL);
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (entry->has_new)
+ {
+ oldcxt = MemoryContextSwitchTo(TopTransactionContext);
+
+ new_tuplestore = tuplestore_begin_heap(false, false, work_mem);
+ dest_new = CreateDestReceiver(DestTuplestore);
+ SetTuplestoreDestReceiverParams(dest_new,
+ new_tuplestore,
+ TopTransactionContext,
+ false,
+ NULL,
+ NULL);
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ /* for all modified tables */
+ foreach(lc, entry->tables)
+ {
+ ListCell *lc2;
+
+ table = (MV_TriggerTable *) lfirst(lc);
+
+ /* loop for self-join */
+ foreach(lc2, table->rte_indexes)
+ {
+ int rte_index = lfirst_int(lc2);
+ TupleDesc tupdesc_old;
+ TupleDesc tupdesc_new;
+
+ /* calculate delta tables */
+ calc_delta(table, rte_index, rewritten, dest_old, dest_new,
+ &tupdesc_old, &tupdesc_new, queryEnv);
+
+ /* Set the table in the query to post-update state */
+ rewritten = rewrite_query_for_postupdate_state(rewritten, table, rte_index);
+
+ PG_TRY();
+ {
+ /* apply the delta tables to the materialized view */
+ apply_delta(matviewOid, old_tuplestore, new_tuplestore,
+ tupdesc_old, tupdesc_new, query);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /* clear view delta tuplestores */
+ if (old_tuplestore)
+ tuplestore_clear(old_tuplestore);
+ if (new_tuplestore)
+ tuplestore_clear(new_tuplestore);
+ }
+ }
+
+ /* Clean up hash entry and delete tuplestores */
+ clean_up_IVM_hash_entry(entry, false);
+ if (old_tuplestore)
+ {
+ dest_old->rDestroy(dest_old);
+ tuplestore_end(old_tuplestore);
+ }
+ if (new_tuplestore)
+ {
+ dest_new->rDestroy(dest_new);
+ tuplestore_end(new_tuplestore);
+ }
+
+ /* Pop the original snapshot. */
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+}
+
+/*
+ * rewrite_query_for_preupdate_state
+ *
+ * Rewrite the query so that base tables' RTEs will represent "pre-update"
+ * state of tables. This is necessary to calculate view delta after multiple
+ * tables are modified.
+ */
+static Query*
+rewrite_query_for_preupdate_state(Query *query, List *tables,
+ ParseState *pstate, Oid matviewid)
+{
+ ListCell *lc;
+ int num_rte = list_length(query->rtable);
+ int i;
+
+
+ /* register delta ENRs */
+ register_delta_ENRs(pstate, query, tables);
+
+ /* XXX: Is necessary? Is this right timing? */
+ AcquireRewriteLocks(query, true, false);
+
+ i = 1;
+ foreach(lc, query->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+
+ ListCell *lc2;
+ foreach(lc2, tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc2);
+ /*
+ * if the modified table is found then replace the original RTE with
+ * "pre-state" RTE and append its index to the list.
+ */
+ if (r->relid == table->table_id)
+ {
+ List *securityQuals;
+ List *withCheckOptions;
+ bool hasRowSecurity;
+ bool hasSubLinks;
+
+ RangeTblEntry *rte_pre = get_prestate_rte(r, table, pstate->p_queryEnv, matviewid);
+
+ /*
+ * Set a row security poslicies of the modified table to the subquery RTE which
+ * represents the pre-update state of the table.
+ */
+ get_row_security_policies(query, table->original_rte, i,
+ &securityQuals, &withCheckOptions,
+ &hasRowSecurity, &hasSubLinks);
+
+ if (hasRowSecurity)
+ {
+ query->hasRowSecurity = true;
+ rte_pre->security_barrier = true;
+ }
+ if (hasSubLinks)
+ query->hasSubLinks = true;
+
+ rte_pre->securityQuals = securityQuals;
+ lfirst(lc) = rte_pre;
+
+ table->rte_indexes = lappend_int(table->rte_indexes, i);
+ break;
+ }
+ }
+
+ /* finish the loop if we processed all RTE included in the original query */
+ if (i++ >= num_rte)
+ break;
+ }
+
+ return query;
+}
+
+/*
+ * register_delta_ENRs
+ *
+ * For all modified tables, make ENRs for their transition tables
+ * and register them to the queryEnv. ENR's RTEs are also appended
+ * into the list in query tree.
+ */
+static void
+register_delta_ENRs(ParseState *pstate, Query *query, List *tables)
+{
+ QueryEnvironment *queryEnv = pstate->p_queryEnv;
+ ListCell *lc;
+ RangeTblEntry *rte;
+
+ foreach(lc, tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc);
+ ListCell *lc2;
+ int count;
+
+ count = 0;
+ foreach(lc2, table->old_tuplestores)
+ {
+ Tuplestorestate *oldtable = (Tuplestorestate *) lfirst(lc2);
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+ ParseNamespaceItem *nsitem;
+
+ enr->md.name = make_delta_enr_name("old", table->table_id, count);
+ enr->md.reliddesc = table->table_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(oldtable);
+ enr->reldata = oldtable;
+ register_ENR(queryEnv, enr);
+
+ nsitem = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ rte = nsitem->p_rte;
+
+ query->rtable = lappend(query->rtable, rte);
+
+ count++;
+ }
+
+ count = 0;
+ foreach(lc2, table->new_tuplestores)
+ {
+ Tuplestorestate *newtable = (Tuplestorestate *) lfirst(lc2);
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+ ParseNamespaceItem *nsitem;
+
+ enr->md.name = make_delta_enr_name("new", table->table_id, count);
+ enr->md.reliddesc = table->table_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(newtable);
+ enr->reldata = newtable;
+ register_ENR(queryEnv, enr);
+
+ nsitem = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ rte = nsitem->p_rte;
+
+ query->rtable = lappend(query->rtable, rte);
+
+ count++;
+ }
+ }
+}
+
+#define DatumGetItemPointer(X) ((ItemPointer) DatumGetPointer(X))
+#define PG_GETARG_ITEMPOINTER(n) DatumGetItemPointer(PG_GETARG_DATUM(n))
+
+/*
+ * ivm_visible_in_prestate
+ *
+ * Check visibility of a tuple specified by the tableoid and item pointer
+ * using the snapshot taken just before the table was modified.
+ */
+Datum
+ivm_visible_in_prestate(PG_FUNCTION_ARGS)
+{
+ Oid tableoid = PG_GETARG_OID(0);
+ ItemPointer itemPtr = PG_GETARG_ITEMPOINTER(1);
+ Oid matviewOid = PG_GETARG_OID(2);
+ MV_TriggerHashEntry *entry;
+ MV_TriggerTable *table = NULL;
+ ListCell *lc;
+ bool found;
+ bool result;
+
+ if (!in_delta_calculation)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("ivm_visible_in_prestate can be called only in delta calculation")));
+
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_FIND, &found);
+ Assert (found && entry != NULL);
+
+ foreach(lc, entry->tables)
+ {
+ table = (MV_TriggerTable *) lfirst(lc);
+ if (table->table_id == tableoid)
+ break;
+ }
+
+ Assert (table != NULL);
+
+ result = table_tuple_fetch_row_version(table->rel, itemPtr, entry->snapshot, table->slot);
+
+ PG_RETURN_BOOL(result);
+}
+
+/*
+ * get_prestate_rte
+ *
+ * Rewrite RTE of the modified table to a subquery which represents
+ * "pre-state" table. The original RTE is saved in table->rte_original.
+ */
+static RangeTblEntry*
+get_prestate_rte(RangeTblEntry *rte, MV_TriggerTable *table,
+ QueryEnvironment *queryEnv, Oid matviewid)
+{
+ StringInfoData str;
+ RawStmt *raw;
+ Query *subquery;
+ Relation rel;
+ ParseState *pstate;
+ char *relname;
+ int i;
+
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * We can use NoLock here since AcquireRewriteLocks should
+ * have locked the relation already.
+ */
+ rel = table_open(table->table_id, NoLock);
+ relname = quote_qualified_identifier(
+ get_namespace_name(RelationGetNamespace(rel)),
+ RelationGetRelationName(rel));
+ table_close(rel, NoLock);
+
+ /*
+ * Filtering inserted row using the snapshot taken before the table
+ * is modified. ctid is required for maintaining outer join views.
+ */
+ initStringInfo(&str);
+ appendStringInfo(&str,
+ "SELECT t.* FROM %s t"
+ " WHERE pg_catalog.ivm_visible_in_prestate(t.tableoid, t.ctid ,%d::pg_catalog.oid)",
+ relname, matviewid);
+
+ /*
+ * Append deleted rows contained in old transition tables.
+ */
+ for (i = 0; i < list_length(table->old_tuplestores); i++)
+ {
+ appendStringInfo(&str, " UNION ALL ");
+ appendStringInfo(&str," SELECT * FROM %s",
+ make_delta_enr_name("old", table->table_id, i));
+ }
+
+ /* Get a subquery representing pre-state of the table */
+ raw = (RawStmt*)linitial(raw_parser(str.data, RAW_PARSE_DEFAULT));
+ subquery = transformStmt(pstate, raw->stmt);
+
+ /* save the original RTE */
+ table->original_rte = copyObject(rte);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = subquery;
+ rte->security_barrier = false;
+
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->perminfoindex = 0; /* no permission checking for this RTE */
+ rte->inh = false; /* must not be set for a subquery */
+
+ return rte;
+}
+
+/*
+ * make_delta_enr_name
+ *
+ * Make a name for ENR of a transition table from the base table's oid.
+ * prefix will be "new" or "old" depending on its transition table kind..
+ */
+static char*
+make_delta_enr_name(const char *prefix, Oid relid, int count)
+{
+ char buf[NAMEDATALEN];
+ char *name;
+
+ snprintf(buf, NAMEDATALEN, "__ivm_%s_%u_%u", prefix, relid, count);
+ name = pstrdup(buf);
+
+ return name;
+}
+
+/*
+ * replace_rte_with_delta
+ *
+ * Replace RTE of the modified table with a single table delta that combine its
+ * all transition tables.
+ */
+static RangeTblEntry*
+replace_rte_with_delta(RangeTblEntry *rte, MV_TriggerTable *table, bool is_new,
+ QueryEnvironment *queryEnv)
+{
+ Oid relid = table->table_id;
+ StringInfoData str;
+ ParseState *pstate;
+ RawStmt *raw;
+ Query *sub;
+ int num_tuplestores = list_length(is_new ? table->new_tuplestores : table->old_tuplestores);
+ int i;
+
+ /* the previous RTE must be a subquery which represents "pre-state" table */
+ Assert(rte->rtekind == RTE_SUBQUERY);
+
+ /* Create a ParseState for rewriting the view definition query */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ initStringInfo(&str);
+
+ for (i = 0; i < num_tuplestores; i++)
+ {
+ if (i > 0)
+ appendStringInfo(&str, " UNION ALL ");
+
+ appendStringInfo(&str,
+ " SELECT * FROM %s",
+ make_delta_enr_name(is_new ? "new" : "old", relid, i));
+ }
+
+ raw = (RawStmt*)linitial(raw_parser(str.data, RAW_PARSE_DEFAULT));
+ sub = transformStmt(pstate, raw->stmt);
+
+ /*
+ * Update the subquery so that it represent the combined transition
+ * table. Note that we leave the security_barrier and securityQuals
+ * fields so that the subquery relation can be protected by the RLS
+ * policy as same as the modified table.
+ */
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+
+ return rte;
+}
+
+/*
+ * rewrite_query_for_counting
+ *
+ * Rewrite query for counting duplicated tuples.
+ */
+static Query *
+rewrite_query_for_counting(Query *query, ParseState *pstate)
+{
+ TargetEntry *tle_count;
+ FuncCall *fn;
+ Node *node;
+
+ /* Add count(*) for counting distinct tuples in views */
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+ if (!query->groupClause && !query->hasAggs)
+ query->groupClause = transformDistinctClause(NULL, &query->targetList, query->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle_count = makeTargetEntry((Expr *) node,
+ list_length(query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ query->targetList = lappend(query->targetList, tle_count);
+ query->hasAggs = true;
+
+ return query;
+}
+
+/*
+ * calc_delta
+ *
+ * Calculate view deltas generated under the modification of a table specified
+ * by the RTE index.
+ */
+static void
+calc_delta(MV_TriggerTable *table, int rte_index, Query *query,
+ DestReceiver *dest_old, DestReceiver *dest_new,
+ TupleDesc *tupdesc_old, TupleDesc *tupdesc_new,
+ QueryEnvironment *queryEnv)
+{
+ ListCell *lc = list_nth_cell(query->rtable, rte_index - 1);
+ RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc);
+
+ in_delta_calculation = true;
+
+ /* Generate old delta */
+ if (list_length(table->old_tuplestores) > 0)
+ {
+ /* Replace the modified table with the old delta table and calculate the old view delta. */
+ replace_rte_with_delta(rte, table, false, queryEnv);
+ refresh_matview_datafill(dest_old, query, queryEnv, tupdesc_old, "");
+ }
+
+ /* Generate new delta */
+ if (list_length(table->new_tuplestores) > 0)
+ {
+ /* Replace the modified table with the new delta table and calculate the new view delta*/
+ replace_rte_with_delta(rte, table, true, queryEnv);
+ refresh_matview_datafill(dest_new, query, queryEnv, tupdesc_new, "");
+ }
+
+ in_delta_calculation = false;
+}
+
+/*
+ * rewrite_query_for_postupdate_state
+ *
+ * Rewrite the query so that the specified base table's RTEs will represent
+ * "post-update" state of tables. This is called after the view delta
+ * calculation due to changes on this table finishes.
+ */
+static Query*
+rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte_index)
+{
+ ListCell *lc = list_nth_cell(query->rtable, rte_index - 1);
+
+ /* Retore the original RTE */
+ lfirst(lc) = table->original_rte;
+
+ return query;
+}
+
+/*
+ * apply_delta
+ *
+ * Apply deltas to the materialized view. In outer join cases, this requires
+ * the view maintenance graph.
+ */
+static void
+apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
+ TupleDesc tupdesc_old, TupleDesc tupdesc_new,
+ Query *query)
+{
+ StringInfoData querybuf;
+ StringInfoData target_list_buf;
+ Relation matviewRel;
+ char *matviewname;
+ ListCell *lc;
+ int i;
+ List *keys = NIL;
+
+
+ /*
+ * get names of the materialized view and delta tables
+ */
+
+ matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ /*
+ * Build parts of the maintenance queries
+ */
+
+ initStringInfo(&querybuf);
+ initStringInfo(&target_list_buf);
+
+ /* build string of target list */
+ for (i = 0; i < matviewRel->rd_att->natts; i++)
+ {
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+ char *resname = NameStr(attr->attname);
+
+ if (i != 0)
+ appendStringInfo(&target_list_buf, ", ");
+ appendStringInfo(&target_list_buf, "%s", quote_qualified_identifier(NULL, resname));
+ }
+
+ i = 0;
+ foreach (lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+
+ i++;
+
+ if (tle->resjunk)
+ continue;
+
+ keys = lappend(keys, attr);
+ }
+
+ /* Start maintaining the materialized view. */
+ OpenMatViewIncrementalMaintenance();
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* For tuple deletion */
+ if (old_tuplestores && tuplestore_tuple_count(old_tuplestores) > 0)
+ {
+ EphemeralNamedRelation enr = palloc(sizeof(EphemeralNamedRelationData));
+ int rc;
+
+ /* convert tuplestores to ENR, and register for SPI */
+ enr->md.name = pstrdup(OLD_DELTA_ENRNAME);
+ enr->md.reliddesc = InvalidOid;
+ enr->md.tupdesc = tupdesc_old;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(old_tuplestores);
+ enr->reldata = old_tuplestores;
+
+ rc = SPI_register_relation(enr);
+ if (rc != SPI_OK_REL_REGISTER)
+ elog(ERROR, "SPI_register failed");
+
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+
+ }
+ /* For tuple insertion */
+ if (new_tuplestores && tuplestore_tuple_count(new_tuplestores) > 0)
+ {
+ EphemeralNamedRelation enr = palloc(sizeof(EphemeralNamedRelationData));
+ int rc;
+
+ /* convert tuplestores to ENR, and register for SPI */
+ enr->md.name = pstrdup(NEW_DELTA_ENRNAME);
+ enr->md.reliddesc = InvalidOid;
+ enr->md.tupdesc = tupdesc_new;;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(new_tuplestores);
+ enr->reldata = new_tuplestores;
+
+ rc = SPI_register_relation(enr);
+ if (rc != SPI_OK_REL_REGISTER)
+ elog(ERROR, "SPI_register failed");
+
+ /* apply new delta */
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ table_close(matviewRel, NoLock);
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
+
+/*
+ * apply_old_delta
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is not required.
+ */
+static void
+apply_old_delta(const char *matviewname, const char *deltaname_old,
+ List *keys)
+{
+ StringInfoData querybuf;
+ StringInfoData keysbuf;
+ char *match_cond;
+ ListCell *lc;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&keysbuf);
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&keysbuf, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&keysbuf, ", ");
+ }
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "DELETE FROM %s WHERE ctid IN ("
+ "SELECT tid FROM (SELECT pg_catalog.row_number() over (partition by %s) AS \"__ivm_row_number__\","
+ "mv.ctid AS tid,"
+ "diff.\"__ivm_count__\""
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s) v "
+ "WHERE v.\"__ivm_row_number__\" OPERATOR(pg_catalog.<=) v.\"__ivm_count__\")",
+ matviewname,
+ keysbuf.data,
+ matviewname, deltaname_old,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
+/*
+ * apply_new_delta
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is not required.
+ */
+static void
+apply_new_delta(const char *matviewname, const char *deltaname_new,
+ StringInfo target_list)
+{
+ StringInfoData querybuf;
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "INSERT INTO %s (%s) SELECT %s FROM ("
+ "SELECT diff.*, pg_catalog.generate_series(1, diff.\"__ivm_count__\")"
+ " AS __ivm_generate_series__ "
+ "FROM %s AS diff) AS v",
+ matviewname, target_list->data, target_list->data,
+ deltaname_new);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
+/*
+ * get_matching_condition_string
+ *
+ * Build a predicate string for looking for a tuple with given keys.
+ */
+static char *
+get_matching_condition_string(List *keys)
+{
+ StringInfoData match_cond;
+ ListCell *lc;
+
+ /* If there is no key columns, the condition is always true. */
+ if (keys == NIL)
+ return "true";
+
+ initStringInfo(&match_cond);
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ char *mv_resname = quote_qualified_identifier("mv", resname);
+ char *diff_resname = quote_qualified_identifier("diff", resname);
+ Oid typid = attr->atttypid;
+
+ /* Considering NULL values, we can not use simple = operator. */
+ appendStringInfo(&match_cond, "(");
+ generate_equal(&match_cond, typid, mv_resname, diff_resname);
+ appendStringInfo(&match_cond, " OR (%s IS NULL AND %s IS NULL))",
+ mv_resname, diff_resname);
+
+ if (lnext(keys, lc))
+ appendStringInfo(&match_cond, " AND ");
+ }
+
+ return match_cond.data;
+}
+
+/*
+ * generate_equals
+ *
+ * Generate an equality clause using given operands' default equality
+ * operator.
+ */
+static void
+generate_equal(StringInfo querybuf, Oid opttype,
+ const char *leftop, const char *rightop)
+{
+ TypeCacheEntry *typentry;
+
+ typentry = lookup_type_cache(opttype, TYPECACHE_EQ_OPR);
+ if (!OidIsValid(typentry->eq_opr))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FUNCTION),
+ errmsg("could not identify an equality operator for type %s",
+ format_type_be_qualified(opttype))));
+
+ generate_operator_clause(querybuf,
+ leftop, opttype,
+ typentry->eq_opr,
+ rightop, opttype);
+}
+
+/*
+ * mv_InitHashTables
+ */
+static void
+mv_InitHashTables(void)
+{
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(Oid);
+ ctl.entrysize = sizeof(MV_TriggerHashEntry);
+ mv_trigger_info = hash_create("MV trigger info",
+ MV_INIT_QUERYHASHSIZE,
+ &ctl, HASH_ELEM | HASH_BLOBS);
+}
+
+/*
+ * AtAbort_IVM
+ *
+ * Clean up hash entries for all materialized views. This is called at
+ * transaction abort.
+ */
+void
+AtAbort_IVM()
+{
+ HASH_SEQ_STATUS seq;
+ MV_TriggerHashEntry *entry;
+
+ if (mv_trigger_info)
+ {
+ hash_seq_init(&seq, mv_trigger_info);
+ while ((entry = hash_seq_search(&seq)) != NULL)
+ clean_up_IVM_hash_entry(entry, true);
+ }
+ in_delta_calculation = false;
+}
+
+/*
+ * clean_up_IVM_hash_entry
+ *
+ * Clean up tuple stores and hash entries for a materialized view after its
+ * maintenance finished.
+ */
+static void
+clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry, bool is_abort)
+{
+ bool found;
+ ListCell *lc;
+
+ foreach(lc, entry->tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc);
+
+ list_free(table->old_tuplestores);
+ list_free(table->new_tuplestores);
+ if (!is_abort)
+ {
+ ExecDropSingleTupleTableSlot(table->slot);
+ table_close(table->rel, NoLock);
+ }
+ }
+ list_free(entry->tables);
+
+ if (!is_abort)
+ UnregisterSnapshot(entry->snapshot);
+
+ hash_search(mv_trigger_info, (void *) &entry->matview_id, HASH_REMOVE, &found);
+}
+
+/*
+ * isIvmName
+ *
+ * Check if this is a IVM hidden column from the name.
+ */
+bool
+isIvmName(const char *s)
+{
+ if (s)
+ return (strncmp(s, "__ivm_", 6) == 0);
+ return false;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9805bc6118..f6896d77b4 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12062,4 +12062,14 @@
proname => 'any_value_transfn', prorettype => 'anyelement',
proargtypes => 'anyelement anyelement', prosrc => 'any_value_transfn' },
+# IVM
+{ oid => '786', descr => 'ivm trigger (before)',
+ proname => 'IVM_immediate_before', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_before' },
+{ oid => '787', descr => 'ivm trigger (after)',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+{ oid => '788', descr => 'ivm filetring ',
+ proname => 'ivm_visible_in_prestate', provolatile => 's', prorettype => 'bool',
+ proargtypes => 'oid tid oid', prosrc => 'ivm_visible_in_prestate' },
]
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 3647f96f73..09a64fa2e5 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -16,6 +16,7 @@
#include "catalog/objectaddress.h"
#include "nodes/params.h"
+#include "nodes/pathnodes.h"
#include "parser/parse_node.h"
#include "tcop/dest.h"
#include "utils/queryenvironment.h"
@@ -25,6 +26,9 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
ParamListInfo params, QueryEnvironment *queryEnv,
QueryCompletion *qc);
+extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
+extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index 9eaa6212a1..504b83a446 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -15,6 +15,7 @@
#define MATVIEW_H
#include "catalog/objectaddress.h"
+#include "fmgr.h"
#include "nodes/params.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
@@ -23,6 +24,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
ParamListInfo params, QueryCompletion *qc);
@@ -30,4 +33,10 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid transientoid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_before(PG_FUNCTION_ARGS);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+extern Datum IVM_visible_in_prestate(PG_FUNCTION_ARGS);
+extern void AtAbort_IVM(void);
+extern bool isIvmName(const char *s);
+
#endif /* MATVIEW_H */
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0007-Add-DISTINCT-support-for-IVM.patch"
Content-Disposition: attachment;
filename="v29-0007-Add-DISTINCT-support-for-IVM.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 36+ messages in thread
end of thread, other threads:[~2023-05-31 09:59 UTC | newest]
Thread overview: 36+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-10-26 01:07 [PATCH 6/6] Check removal of in-reading segment file. Kyotaro Horiguchi <[email protected]>
2023-01-11 21:58 Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 02:19 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 05:23 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-01-13 18:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-13 23:36 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 17:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-17 19:12 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-17 22:00 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-19 11:17 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) vignesh C <[email protected]>
2023-01-19 21:28 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-20 02:15 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-01-24 08:22 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 08:25 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-01-24 22:35 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-25 07:56 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-02-08 06:38 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-09 05:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-11 18:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-14 19:08 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-02-22 01:50 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-02-22 04:09 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Justin Pryzby <[email protected]>
2023-03-04 23:21 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-03-06 06:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-03-06 06:48 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Kyotaro Horiguchi <[email protected]>
2023-03-06 15:09 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-02-26 18:20 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-26 18:52 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 14:24 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-02-27 15:30 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 19:03 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Melanie Plageman <[email protected]>
2023-02-27 19:58 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Tom Lane <[email protected]>
2023-02-27 23:18 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Andres Freund <[email protected]>
2023-01-18 06:10 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Maciek Sakrejda <[email protected]>
2023-01-16 21:41 ` Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?) Maciek Sakrejda <[email protected]>
2023-05-31 09:59 [PATCH v29 06/11] Add Incremental View Maintenance support Yugo Nagata <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox