agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feed[PATCH 17/17] User and development documentation.
28+ messages / 8 participants
[nested] [flat]
* [PATCH 17/17] User and development documentation.
@ 2019-07-05 14:24 Antonin Houska <ah@cybertec.at>
0 siblings, 0 replies; 28+ messages in thread
From: Antonin Houska @ 2019-07-05 14:24 UTC (permalink / raw)
---
doc/src/sgml/config.sgml | 14 ++
doc/src/sgml/encryption.sgml | 95 ++++++++++++++
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/installation.sgml | 14 +-
doc/src/sgml/postgres.sgml | 1 +
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/initdb.sgml | 11 ++
doc/src/sgml/ref/pg_checksums.sgml | 13 +-
doc/src/sgml/ref/pg_ctl-ref.sgml | 11 ++
doc/src/sgml/ref/pg_keytool.sgml | 121 +++++++++++++++++
doc/src/sgml/ref/pg_resetwal.sgml | 11 ++
doc/src/sgml/ref/pg_rewind.sgml | 11 ++
doc/src/sgml/ref/pg_waldump.sgml | 11 ++
doc/src/sgml/ref/pgupgrade.sgml | 31 +++++
doc/src/sgml/reference.sgml | 1 +
src/backend/storage/file/README.encryption | 202 +++++++++++++++++++++++++++++
16 files changed, 542 insertions(+), 7 deletions(-)
create mode 100644 doc/src/sgml/encryption.sgml
create mode 100644 doc/src/sgml/ref/pg_keytool.sgml
create mode 100644 src/backend/storage/file/README.encryption
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c91e3e1550..5d64736a03 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -9004,6 +9004,20 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir'
</listitem>
</varlistentry>
+ <varlistentry id="guc-data-encryption" xreflabel="data_encryption">
+ <term><varname>data_encryption</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>data_encryption</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Reports whether data encryption is enabled for this cluster.
+ See <xref linkend="encryption"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-data-directory-mode" xreflabel="data_directory_mode">
<term><varname>data_directory_mode</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/encryption.sgml b/doc/src/sgml/encryption.sgml
new file mode 100644
index 0000000000..5913883bc0
--- /dev/null
+++ b/doc/src/sgml/encryption.sgml
@@ -0,0 +1,95 @@
+<!-- doc/src/sgml/encryption.sgml -->
+
+<chapter id="encryption">
+ <title>Transparent Cluster Encryption</title>
+
+ <para>
+ Cluster encryption can be used if <acronym>DBA</acronym> cannot or does not
+ want to rely on the filesystem in terms of data confidentiality. If this
+ feature is enabled, <productname>PostgreSQL</productname> encrypts data
+ (both relations and write-ahead-log) when writing it to disk, and decrypts
+ when reading it. The encryption is transparent, so applications see no
+ difference between encrypted and unencrypted cluster.
+ </para>
+
+ <para>
+ To create encrypted cluster, use the <option>-K</option> option to pass the
+ <xref linkend="app-initdb"/> utility path to the command that will retrieve
+ the encryption key. For example:
+<screen>
+<prompt>$</prompt> <userinput>initdb -D /usr/local/pgsql/data -K /usr/local/pgsql/fetch_key_cmd</userinput>
+</screen>
+ Here <filename>/usr/local/pgsql/fetch_key_cmd</filename> is an executable
+ file that writes the encryption key to its standard output and returns
+ zero. The simplest command looks like:
+
+<programlisting>
+echo <your key>
+</programlisting>
+ </para>
+
+ <para>
+ The encryption key is expected in hexadecimal format, two characters
+ (hexadecimal digits) per
+ byte. Since <productname>PostgreSQL</productname> currently uses key
+ of length 32 bytes (256 bits), the expected length of the key string
+ is 64 characters. For example:
+ <computeroutput>
+ 882fb7c12e80280fd664c69d2d636913e86c381ba487c82f77653c0fac8ffc69
+ </computeroutput>
+ </para>
+
+ <para>
+ If the DBA wants to enter password, he can use
+ <xref linkend="app-pg-keytool"/> utility to derive ecryption key from
+ it. For example, if the <option>-K</option> option looks like in the
+ following example, the DBA will be asked for password interactively:
+
+<screen>
+<prompt>$</prompt> <userinput>initdb -K 'read -sp "Cluster encryption password: " PGENCRPWD; echo $PGENCRPWD
+| pg_keytool -D %D -w' -D data
+</userinput>
+</screen>
+
+ </para>
+
+ <para>
+ As long as the application is aware of the data directory, you can
+ put <literal>%D</literal> into the command and the application will replace
+ it with the actual directory path.
+ </para>
+
+ <para>
+ Likewise, the same command can be passed to <xref linkend="app-pg-ctl"/> in
+ order to start the cluster:
+
+<screen>
+<prompt>$</prompt> <userinput>pg_ctl -K 'read -sp "Cluster encryption password: " PGENCRPWD; echo $PGENCRPWD
+| pg_keytool -D %D -w' -D data start
+</userinput>
+</screen>
+ </para>
+
+ <para>
+ If encryption is enabled, <xref linkend="guc-full-page-writes"/> must be
+ turned on, otherwise the server refuses to start. This is because the
+ encryption introduces dependencies between data within a page, and thus
+ server crash during disk write can result in more serious damage of the page
+ than it would do without encryption. The whole page needs to be retrieved
+ from WAL in such a case to ensure reliable recovery.
+ </para>
+
+ <para>
+ Once the <productname>PostgreSQL</productname> server is running, client
+ applications should recognize no difference from an unencrypted cluster,
+ except that <xref linkend="guc-data-encryption"/> variable is set.
+ </para>
+
+ <para>
+ Since WAL is encrypted, any replication solution based on log shipping
+ (<xref linkend="warm-standby"/>) assume that all standby servers are
+ encrypted using the same key as their primary server. On the other hand,
+ <xref linkend="logical-replication"/> allows replication between encrypted
+ and unencrypted clusters, or between clusters encrypted with different keys.
+ </para>
+</chapter>
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 3da2365ea9..d21dabf03b 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -48,6 +48,7 @@
<!ENTITY wal SYSTEM "wal.sgml">
<!ENTITY logical-replication SYSTEM "logical-replication.sgml">
<!ENTITY jit SYSTEM "jit.sgml">
+<!ENTITY encryption SYSTEM "encryption.sgml">
<!-- programmer's guide -->
<!ENTITY bgworker SYSTEM "bgworker.sgml">
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 449386243b..a2daf33e6f 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -245,8 +245,9 @@ su - postgres
<listitem>
<para>
You need <productname>OpenSSL</productname>, if you want to support
- encrypted client connections. <productname>OpenSSL</productname> is
- also required for random number generation on platforms that do not
+ on-disk data encryption or encrypted client
+ connections. <productname>OpenSSL</productname> is also required for
+ random number generation on platforms that do not
have <filename>/dev/urandom</filename> (except Windows). The minimum
version required is 0.9.8.
</para>
@@ -835,10 +836,11 @@ su - postgres
</term>
<listitem>
<para>
- Build with support for <acronym>SSL</acronym> (encrypted)
- connections. This requires the <productname>OpenSSL</productname>
- package to be installed. <filename>configure</filename> will check
- for the required header files and libraries to make sure that
+ Build with support for on-disk data encryption
+ or <acronym>SSL</acronym> (encrypted) connections. This requires
+ the <productname>OpenSSL</productname> package to be
+ installed. <filename>configure</filename> will check for the
+ required header files and libraries to make sure that
your <productname>OpenSSL</productname> installation is sufficient
before proceeding.
</para>
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 3e115f1c76..b0d35600b2 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -163,6 +163,7 @@
&wal;
&logical-replication;
&jit;
+ &encryption;
®ress;
</part>
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 8d91f3529e..a60b02e23f 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -206,6 +206,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgDump SYSTEM "pg_dump.sgml">
<!ENTITY pgDumpall SYSTEM "pg_dumpall.sgml">
<!ENTITY pgIsready SYSTEM "pg_isready.sgml">
+<!ENTITY pgKeytool SYSTEM "pg_keytool.sgml">
<!ENTITY pgReceivewal SYSTEM "pg_receivewal.sgml">
<!ENTITY pgRecvlogical SYSTEM "pg_recvlogical.sgml">
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml
index 7fc3152c6d..ce19151521 100644
--- a/doc/src/sgml/ref/initdb.sgml
+++ b/doc/src/sgml/ref/initdb.sgml
@@ -209,6 +209,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry id="app-initdb-data-encr-cmd" xreflabel="data encryption">
+ <term><option>-K</option></term>
+ <term><option>--encryption-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ Encrypt the cluster data using a key retrieved from the command
+ specified here. See <xref linkend="encryption"/> for details.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="app-initdb-data-checksums" xreflabel="data checksums">
<term><option>-k</option></term>
<term><option>--data-checksums</option></term>
diff --git a/doc/src/sgml/ref/pg_checksums.sgml b/doc/src/sgml/ref/pg_checksums.sgml
index 33706d1d97..7a2cae5e57 100644
--- a/doc/src/sgml/ref/pg_checksums.sgml
+++ b/doc/src/sgml/ref/pg_checksums.sgml
@@ -106,7 +106,18 @@ PostgreSQL documentation
<listitem>
<para>
Only validate checksums in the relation with filenode
- <replaceable>filenode</replaceable>.
+ <replaceable>filenode</replaceable>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-K</option></term>
+ <term><option>--encryption-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ If the cluster is encrypted, run this command to retrieve the
+ encryption key. See <xref linkend="encryption"/> for details.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_ctl-ref.sgml b/doc/src/sgml/ref/pg_ctl-ref.sgml
index e31275a04e..48966eb58e 100644
--- a/doc/src/sgml/ref/pg_ctl-ref.sgml
+++ b/doc/src/sgml/ref/pg_ctl-ref.sgml
@@ -293,6 +293,17 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
+ <term><option>-K</option></term>
+ <term><option>--encryption-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ If the cluster is encrypted, run this command to retrieve the
+ encryption key. See <xref linkend="encryption"/> for details.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-l <replaceable class="parameter">filename</replaceable></option></term>
<term><option>--log=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_keytool.sgml b/doc/src/sgml/ref/pg_keytool.sgml
new file mode 100644
index 0000000000..3c15910885
--- /dev/null
+++ b/doc/src/sgml/ref/pg_keytool.sgml
@@ -0,0 +1,121 @@
+<!--
+doc/src/sgml/ref/pg_keytool.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pg-keytool">
+ <indexterm zone="app-pg-keytool">
+ <primary>pg_keysetup</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_keytool</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_keytool</refname>
+ <refpurpose>produce cluster encryption key, possibly deriving it from a
+ passphrase</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_keysetup</command>
+ <arg rep="repeat"><replaceable class="parameter">option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_keytool</application> reads encryption key or password from
+ the standard input. If it received password, it runs <firstterm>key
+ derivation function</firstterm> on it in order to derive the key. The
+ encryption key is written either to the standard output or sent to via
+ frontend/backend protocol to postmaster process which is being started up.
+ </para>
+
+ <para>
+ One common use case is that <application>pg_keytool</application> is used
+ within encryption key command, see the examples in
+ <xref linkend="encryption"/>. In this case it sends the key to the
+ standard output.
+ </para>
+
+ <para>
+ The other use case is that <productname>PostgreSQL</productname> is
+ started other way than using <xref linkend="app-pg-ctl"/>, typically
+ via <firstterm>systemd</firstterm> or a custom script. In this
+ case <application>pg_keytool</application> can be used to send the key to
+ specified TCP/IP port or to an Unix socket, on which the server
+ listens. This is the same port or socket to which applications will
+ eventually connect, but when the encrypted cluster starts up, it only
+ accepts the key.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable>directory</replaceable></option></term>
+ <listitem>
+ <para>
+ Specifies the directory where the database cluster is
+ stored. Different directory generally means that a different key will
+ be derived from the same password.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-h <replaceable class="parameter">hostname</replaceable></option></term>
+ <term><option>--host=<replaceable class="parameter">hostname</replaceable></option></term>
+ <listitem>
+ <para>
+ Specifies the host name of the machine on which the server is
+ listening. If the value begins with a slash, it is used as the
+ directory for the Unix-domain socket.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-p <replaceable class="parameter">port</replaceable></option></term>
+ <listitem>
+ <para>
+ Specifies the TCP port or the local Unix-domain socket file
+ extension on which the server is listening for
+ connections. Defaults to the value of the <envar>PGPORT</envar>
+ environment variable or, if not set, to the port specified at
+ compile time, usually 5432.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-s</option></term>
+ <listitem>
+ <para>
+ Send the key to <productname>PostgreSQL</productname> server rather
+ than to standard output.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-w</option></term>
+ <listitem>
+ <para>
+ Expect password on standard input rather than encryption key.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </refsect1>
+</refentry>
diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml
index 8a9e22d050..613d14e1cc 100644
--- a/doc/src/sgml/ref/pg_resetwal.sgml
+++ b/doc/src/sgml/ref/pg_resetwal.sgml
@@ -100,6 +100,17 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
+ <term><option>-K</option></term>
+ <term><option>--encryption-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ If the cluster is encrypted, run this command to retrieve the
+ encryption key. See <xref linkend="encryption"/> for details.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-n</option></term>
<term><option>--dry-run</option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
index 4d91eeb0ff..0e4f79b652 100644
--- a/doc/src/sgml/ref/pg_rewind.sgml
+++ b/doc/src/sgml/ref/pg_rewind.sgml
@@ -140,6 +140,17 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
+ <term><option>-K</option></term>
+ <term><option>--encryption-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ If the cluster is encrypted, run this command to retrieve the
+ encryption key. See <xref linkend="encryption"/> for details.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>--source-pgdata=<replaceable class="parameter">directory</replaceable></option></term>
<listitem>
<para>
diff --git a/doc/src/sgml/ref/pg_waldump.sgml b/doc/src/sgml/ref/pg_waldump.sgml
index 329c10e430..ca1e4e1468 100644
--- a/doc/src/sgml/ref/pg_waldump.sgml
+++ b/doc/src/sgml/ref/pg_waldump.sgml
@@ -101,6 +101,17 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
+ <term><option>-K</option></term>
+ <term><option>--encryption-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ If the cluster is encrypted, run this command to retrieve the
+ encryption key. See <xref linkend="encryption"/> for details.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-n <replaceable>limit</replaceable></option></term>
<term><option>--limit=<replaceable>limit</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml
index 82886760f1..0c04990d51 100644
--- a/doc/src/sgml/ref/pgupgrade.sgml
+++ b/doc/src/sgml/ref/pgupgrade.sgml
@@ -120,6 +120,37 @@
</varlistentry>
<varlistentry>
+ <term><option>-K</option></term>
+ <term><option>--encryption-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ If the clusters are encrypted, run this command to retrieve the
+ encryption key. This is the same command that was passed to
+ <xref linkend="app-initdb"/> for cluster creation. For the upgrade to
+ be successful, both old and new clusters must use the same key.
+ Therefore, unless the command contains the <literal>%D</literal>
+ string, it does not matter which cluster the command comes from.
+ </para>
+
+ <note>
+ <para>
+ If you are using <xref linkend="app-pg-keytool"/> to derive the
+ encryption key (see <xref linkend="app-initdb-data-encr-cmd"/>),
+ make sure the <option>-D</option> points to
+ the <emphasis>old</emphasis> cluster. If it pointed to the new
+ one then the new cluster would derive a different key from the same
+ password and thus <xref linkend="pgupgrade"/> would fail to start
+ the new cluster.
+ </para>
+ </note>
+
+ <para>
+ See <xref linkend="encryption"/> for details on the command.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-k</option></term>
<term><option>--link</option></term>
<listitem><para>use hard links instead of copying files to the new
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index cef09dd38b..5eed489f27 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -279,6 +279,7 @@
&pgChecksums;
&pgControldata;
&pgCtl;
+ &pgKeytool;
&pgResetwal;
&pgRewind;
&pgtestfsync;
diff --git a/src/backend/storage/file/README.encryption b/src/backend/storage/file/README.encryption
new file mode 100644
index 0000000000..5ffd587cf5
--- /dev/null
+++ b/src/backend/storage/file/README.encryption
@@ -0,0 +1,202 @@
+src/backend/storage/file/README.encryption
+
+Transparent Cluster Encryption
+==============================
+
+When creating a new cluster (instance), user can choose to have his data
+encrypted on disk. If this feature is active, data is encrypted before it's
+written to disk and decrypted after it has been read, however the data is
+unencrypted in memory (data-at-rest encryption). The following characteristics
+should be considered by anyone who is interested in the feature:
+
+ 1. The encryption is transparent from application's point of view.
+
+ 2. A single key is used to encrypt the whole cluster.
+
+The full instance encryption feature helps to ensure data confidentiality,
+especially when user cannot rely on confidentiality on filesystem level. On
+the other hand, it does not ensure data integrity, i.e. it does not help to
+detect whether an adversary wrote his cipher data to the disk. The block
+cipher methods generally do not protect data integrity, and it'd probably be
+hard anyway because we encrypt postgres data pages (typically of size 8 kB)
+separate from each other. If the attacker only changes part of the page, this
+can be detected if data checksums feature is enabled. And specifically for
+XLOG, each XLOG record has its own checksum.
+
+Since the data is stored on a disk, we naturally base our approach on "Disk
+encryption theory" [1]. For each kind of file we use the AES cipher in the
+appropriate mode of operation. The AES cipher itself encrypts / decrypts
+individual blocks ("encryption blocks") of 16 bytes (128 bits), while the mode
+of operation defines the rules how to apply the cipher to data which spans
+multiple encryption blocks.
+
+Relations
+---------
+
+AES cipher in CBC mode [2] is used to encrypt relation files, one relation
+block (page) at a time. The important characteristic of this mode is that, if
+a single encryption block changes, the next run of the encryption produces an
+encrypted page where not only that block is different, but also all the
+following blocks. Furthermore, as the postgres page starts with LSN, even if
+only the last encryption block of the page changes, the whole cipher text of
+the page will be different after the next encryption.
+
+This propagation of changes across encryption blocks makes it harder for
+adversary to deduce what happens in the database. Without the propagation he
+might be able to see data changes with higher granularity. For example, if
+only the beginning of index page changes, it gives some information on the
+keys inserted.
+
+On the other hand, the same change propagation can cause problems if only a
+part of a new version of encrypted page is written to disk (torn page
+write). If such a page is decrypted during crash recovery, part of the plain
+text can become garbage. This can affect even those parts of the page that
+haven't been modified recently, and therefore XLOG records to fix those parts
+are not available. Therefore, if the cluster encryption is enabled,
+full_page_writes configuration variable must be set too, otherwise the server
+refuses to start. With this setting, the damage described here can be fixed by
+retrieving the full-page-image (FPI) from XLOG and applying it.
+
+However, even if full_page_writes is set, postgres can still avoid writing FPI
+XLOG record if only hint bit(s) changed on a page and if wal_log_hints
+configuration variable is off. The problem with encryption is that even a
+single bit change is propagated to all the following encryption blocks during
+the next encryption of the page. Again, if only part of the new cipher text is
+written, the remaining portion of the page becomes garbage after decryption,
+and we might not be able to fix it during recovery unless we have the FPI in
+the XLOG. So if encryption is enabled, postgres behaves as if wal_log_hints
+was always set.
+
+XLOG
+----
+
+The specific problem of XLOG is that record must not be changed once it has
+been flushed to disk. However if we used a block cipher, and if a new XLOG
+record started in the same encryption block in which the previous record ends,
+that encryption block would become completely different after the next run of
+encryption of the containing page. Torn write of such a block (e.g. if memory
+page boundary crosses the encryption block) is likely to make decryption
+produce garbage, which will also appear in the already-flushed record.
+
+Therefore we encrypt XLOG using a stream cipher, or rather block cipher in
+stream mode of operation. Stream cipher uses XOR operation to combine a "key
+stream" with the input stream, and it does not matter if the length of the
+input stream is aligned to any value. In particular, the CTR mode [3] was
+chosen because it allows for both read and write operations to be
+parallelized.
+
+The XLOG is encrypted / decrypted one XLOG page (typically 8 kB) at a time, so
+rather than a single stream, the encrypted XLOG is actually a sequence of
+per-page streams.
+
+Since stream ciphers are susceptible to "reused key attack" [4], we must
+ensure that the unused part of the last XLOG page (filled with zeroes) is
+never encrypted.
+
+Temporary files
+---------------
+
+BufFileWrite() and BufFileRead() functions (see buffile.c) hide the encryption
+/ decryption from caller. The encryption / decryption processes one buffer at
+a time so that the buffers can be retrieved independent from each other.
+
+If the encryption is enabled, the following requirements need to be taken into
+account:
+
+1. The file buffer cannot be positioned at arbitrary offset of the file. If
+the encryption routine starts at some position of the file, decryption must
+not start elsewhere because there's no way to determine which initialization
+vector was used internally for the corresponding encryption blocks (of 16
+bytes) during encryption. It makes sense to position the buffer at file offset
+that is whole multiple of buffer size.
+
+2. In general, the useful (written) data does not fill whole multiple of
+encryption blocks, but we must write the whole blocks for decryption to
+succeed. This implies that we need to fill the unused part of the last block
+with zeroes and also remember the amount of useful bytes in the segment file.
+(In fact we align the segment file size to file buffers instead of encryption
+blocks, which probably makes the implementation a bit simpler.)
+
+Stream cipher might seem like a solution of the padding problem, but we cannot
+use it here because parts of the temporary file can be rewritten. That would
+expose the temporary file to "reused key attack" [4].
+
+Auxiliary files
+---------------
+
+To store other kinds of data encrypted than the ones above, developers are
+advised to use BufFileWriteTransient() and BufFileReadTransient() functions
+(also located in buffile.c). These are especially useful if some data
+structure should be written to an encrypted file and user does not want to
+care whether the next write position is at encryption block boundary.
+
+Wherever the write starts, BufFileWriteTransient() first reads the whole
+encryption block that contains the start positions, then decrypts it, adds the
+new data to it, encrypts the whole block and writes it back to the
+file. Likewise, BufFileReadTransient() ensures that only the whole encryption
+blocks are read from disk and decrypted, whether the next read position is at
+block boundary or not.
+
+Serialization of data changes during logical decoding (reorderbuffer.c) is the
+typical use case for this API.
+
+Initialization vector (IV), encryption tweak
+--------------------------------------------
+
+Besides the input data and the key, both block and stream cipher used for the
+postgres cluster encryption require an initialization vector. It should make
+analysis of the encrypted data more difficult. Also according to [1], the same
+data should be encrypted differently if located elsewhere on the disk. Term
+"encryption tweak" is more common in the context of disk encryption.
+
+When encrypting relations, each encryption unit (i.e. page) has an unique
+tweak, which consists of RelFileNode structure, buffer number and fork
+number. As a consequence, we need to decrypt page using the existing tweak and
+encrypt it with a new tweak if copying it from one relation to
+another. Typically this happens when a new database is being created from
+template database.
+
+To avoid "reencryption" during pg_upgrade, all the OIDs contained in the
+RelFileNode structure (tablespace OID, database OID and relation file OID)
+must be preserved. If pg_upgrade had to reencrypt files, it could not be used
+with the --link option.
+
+XLOG encryption tweak consists of timeline, segment number and offset at which
+the XLOG page starts in the segment. The "reencryption" takes place when XLOG
+page is copied from one timeline to another, typicially at the end of
+recovery.
+
+As for temporary files, PID of the owning backend, file / fileset number and
+block number (where block is of the same size as a relation page) within the
+file provide sufficient uniqueness, so we use these to generate the tweak.
+
+Auxiliary files can be closed and reopened by another backend, so there's no
+PID strictly associated with them. Therefore we generate the tweak by hashing
+the file path, and appending block number to the hash.
+
+Replication
+-----------
+
+During streaming replication, the walsender process sends the XLOG encrypted
+and walreceiver just writes it. Decryption is performed before the slave
+cluster tries to apply the changes. That implies that the same encryption key
+must be used both master and slave. If it should be possible someday to stream
+the XLOG unencrypted. One implication is that pg_basebackup should then also
+receive the relation data unencrypted.
+
+As for logical replication, the only change introduced by this feature is that
+the XLOG has to be decrypted before the contained data changes can be
+decoded. The data changes are transferred to the subscribing database /
+cluster unencrypted, so both master and slave can use different encryption
+keys.
+
+References
+----------
+
+[1] https://en.wikipedia.org/wiki/Disk_encryption_theory
+
+[2] https://en.wikipedia.org/wiki/Disk_encryption_theory#Cipher-block_chaining_(CBC)
+
+[3] https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_(CTR)
+
+[4] https://en.wikipedia.org/wiki/Stream_cipher_attacks#Reused_key_attack
--
2.13.7
--=-=-=--
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v20 7/8] Row pattern recognition patch (tests).
@ 2024-05-24 02:26 Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 0 replies; 28+ messages in thread
From: Tatsuo Ishii @ 2024-05-24 02:26 UTC (permalink / raw)
---
src/test/regress/expected/rpr.out | 836 +++++++++++++++++++++++++++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/rpr.sql | 405 ++++++++++++++
3 files changed, 1242 insertions(+), 1 deletion(-)
create mode 100644 src/test/regress/expected/rpr.out
create mode 100644 src/test/regress/sql/rpr.sql
diff --git a/src/test/regress/expected/rpr.out b/src/test/regress/expected/rpr.out
new file mode 100644
index 0000000000..0789e09435
--- /dev/null
+++ b/src/test/regress/expected/rpr.out
@@ -0,0 +1,836 @@
+--
+-- Test for row pattern definition clause
+--
+CREATE TEMP TABLE stock (
+ company TEXT,
+ tdate DATE,
+ price INTEGER
+);
+INSERT INTO stock VALUES ('company1', '2023-07-01', 100);
+INSERT INTO stock VALUES ('company1', '2023-07-02', 200);
+INSERT INTO stock VALUES ('company1', '2023-07-03', 150);
+INSERT INTO stock VALUES ('company1', '2023-07-04', 140);
+INSERT INTO stock VALUES ('company1', '2023-07-05', 150);
+INSERT INTO stock VALUES ('company1', '2023-07-06', 90);
+INSERT INTO stock VALUES ('company1', '2023-07-07', 110);
+INSERT INTO stock VALUES ('company1', '2023-07-08', 130);
+INSERT INTO stock VALUES ('company1', '2023-07-09', 120);
+INSERT INTO stock VALUES ('company1', '2023-07-10', 130);
+INSERT INTO stock VALUES ('company2', '2023-07-01', 50);
+INSERT INTO stock VALUES ('company2', '2023-07-02', 2000);
+INSERT INTO stock VALUES ('company2', '2023-07-03', 1500);
+INSERT INTO stock VALUES ('company2', '2023-07-04', 1400);
+INSERT INTO stock VALUES ('company2', '2023-07-05', 1500);
+INSERT INTO stock VALUES ('company2', '2023-07-06', 60);
+INSERT INTO stock VALUES ('company2', '2023-07-07', 1100);
+INSERT INTO stock VALUES ('company2', '2023-07-08', 1300);
+INSERT INTO stock VALUES ('company2', '2023-07-09', 1200);
+INSERT INTO stock VALUES ('company2', '2023-07-10', 1300);
+SELECT * FROM stock;
+ company | tdate | price
+----------+------------+-------
+ company1 | 07-01-2023 | 100
+ company1 | 07-02-2023 | 200
+ company1 | 07-03-2023 | 150
+ company1 | 07-04-2023 | 140
+ company1 | 07-05-2023 | 150
+ company1 | 07-06-2023 | 90
+ company1 | 07-07-2023 | 110
+ company1 | 07-08-2023 | 130
+ company1 | 07-09-2023 | 120
+ company1 | 07-10-2023 | 130
+ company2 | 07-01-2023 | 50
+ company2 | 07-02-2023 | 2000
+ company2 | 07-03-2023 | 1500
+ company2 | 07-04-2023 | 1400
+ company2 | 07-05-2023 | 1500
+ company2 | 07-06-2023 | 60
+ company2 | 07-07-2023 | 1100
+ company2 | 07-08-2023 | 1300
+ company2 | 07-09-2023 | 1200
+ company2 | 07-10-2023 | 1300
+(20 rows)
+
+-- basic test using PREV
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value | nth_second
+----------+------------+-------+-------------+------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 140 | 07-02-2023
+ company1 | 07-02-2023 | 200 | | |
+ company1 | 07-03-2023 | 150 | | |
+ company1 | 07-04-2023 | 140 | | |
+ company1 | 07-05-2023 | 150 | | |
+ company1 | 07-06-2023 | 90 | 90 | 120 | 07-07-2023
+ company1 | 07-07-2023 | 110 | | |
+ company1 | 07-08-2023 | 130 | | |
+ company1 | 07-09-2023 | 120 | | |
+ company1 | 07-10-2023 | 130 | | |
+ company2 | 07-01-2023 | 50 | 50 | 1400 | 07-02-2023
+ company2 | 07-02-2023 | 2000 | | |
+ company2 | 07-03-2023 | 1500 | | |
+ company2 | 07-04-2023 | 1400 | | |
+ company2 | 07-05-2023 | 1500 | | |
+ company2 | 07-06-2023 | 60 | 60 | 1200 | 07-07-2023
+ company2 | 07-07-2023 | 1100 | | |
+ company2 | 07-08-2023 | 1300 | | |
+ company2 | 07-09-2023 | 1200 | | |
+ company2 | 07-10-2023 | 1300 | | |
+(20 rows)
+
+-- basic test using PREV. UP appears twice
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+ UP+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value | nth_second
+----------+------------+-------+-------------+------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 150 | 07-02-2023
+ company1 | 07-02-2023 | 200 | | |
+ company1 | 07-03-2023 | 150 | | |
+ company1 | 07-04-2023 | 140 | | |
+ company1 | 07-05-2023 | 150 | | |
+ company1 | 07-06-2023 | 90 | 90 | 130 | 07-07-2023
+ company1 | 07-07-2023 | 110 | | |
+ company1 | 07-08-2023 | 130 | | |
+ company1 | 07-09-2023 | 120 | | |
+ company1 | 07-10-2023 | 130 | | |
+ company2 | 07-01-2023 | 50 | 50 | 1500 | 07-02-2023
+ company2 | 07-02-2023 | 2000 | | |
+ company2 | 07-03-2023 | 1500 | | |
+ company2 | 07-04-2023 | 1400 | | |
+ company2 | 07-05-2023 | 1500 | | |
+ company2 | 07-06-2023 | 60 | 60 | 1300 | 07-07-2023
+ company2 | 07-07-2023 | 1100 | | |
+ company2 | 07-08-2023 | 1300 | | |
+ company2 | 07-09-2023 | 1200 | | |
+ company2 | 07-10-2023 | 1300 | | |
+(20 rows)
+
+-- basic test using PREV. Use '*'
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP* DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value | nth_second
+----------+------------+-------+-------------+------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 140 | 07-02-2023
+ company1 | 07-02-2023 | 200 | | |
+ company1 | 07-03-2023 | 150 | | |
+ company1 | 07-04-2023 | 140 | | |
+ company1 | 07-05-2023 | 150 | 150 | 90 | 07-06-2023
+ company1 | 07-06-2023 | 90 | | |
+ company1 | 07-07-2023 | 110 | 110 | 120 | 07-08-2023
+ company1 | 07-08-2023 | 130 | | |
+ company1 | 07-09-2023 | 120 | | |
+ company1 | 07-10-2023 | 130 | | |
+ company2 | 07-01-2023 | 50 | 50 | 1400 | 07-02-2023
+ company2 | 07-02-2023 | 2000 | | |
+ company2 | 07-03-2023 | 1500 | | |
+ company2 | 07-04-2023 | 1400 | | |
+ company2 | 07-05-2023 | 1500 | 1500 | 60 | 07-06-2023
+ company2 | 07-06-2023 | 60 | | |
+ company2 | 07-07-2023 | 1100 | 1100 | 1200 | 07-08-2023
+ company2 | 07-08-2023 | 1300 | | |
+ company2 | 07-09-2023 | 1200 | | |
+ company2 | 07-10-2023 | 1300 | | |
+(20 rows)
+
+-- basic test with none greedy pattern
+SELECT company, tdate, price, count(*) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (A A A)
+ DEFINE
+ A AS price >= 140 AND price <= 150
+);
+ company | tdate | price | count
+----------+------------+-------+-------
+ company1 | 07-01-2023 | 100 | 0
+ company1 | 07-02-2023 | 200 | 0
+ company1 | 07-03-2023 | 150 | 3
+ company1 | 07-04-2023 | 140 | 0
+ company1 | 07-05-2023 | 150 | 0
+ company1 | 07-06-2023 | 90 | 0
+ company1 | 07-07-2023 | 110 | 0
+ company1 | 07-08-2023 | 130 | 0
+ company1 | 07-09-2023 | 120 | 0
+ company1 | 07-10-2023 | 130 | 0
+ company2 | 07-01-2023 | 50 | 0
+ company2 | 07-02-2023 | 2000 | 0
+ company2 | 07-03-2023 | 1500 | 0
+ company2 | 07-04-2023 | 1400 | 0
+ company2 | 07-05-2023 | 1500 | 0
+ company2 | 07-06-2023 | 60 | 0
+ company2 | 07-07-2023 | 1100 | 0
+ company2 | 07-08-2023 | 1300 | 0
+ company2 | 07-09-2023 | 1200 | 0
+ company2 | 07-10-2023 | 1300 | 0
+(20 rows)
+
+-- last_value() should remain consistent
+SELECT company, tdate, price, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | last_value
+----------+------------+-------+------------
+ company1 | 07-01-2023 | 100 | 140
+ company1 | 07-02-2023 | 200 |
+ company1 | 07-03-2023 | 150 |
+ company1 | 07-04-2023 | 140 |
+ company1 | 07-05-2023 | 150 |
+ company1 | 07-06-2023 | 90 | 120
+ company1 | 07-07-2023 | 110 |
+ company1 | 07-08-2023 | 130 |
+ company1 | 07-09-2023 | 120 |
+ company1 | 07-10-2023 | 130 |
+ company2 | 07-01-2023 | 50 | 1400
+ company2 | 07-02-2023 | 2000 |
+ company2 | 07-03-2023 | 1500 |
+ company2 | 07-04-2023 | 1400 |
+ company2 | 07-05-2023 | 1500 |
+ company2 | 07-06-2023 | 60 | 1200
+ company2 | 07-07-2023 | 1100 |
+ company2 | 07-08-2023 | 1300 |
+ company2 | 07-09-2023 | 1200 |
+ company2 | 07-10-2023 | 1300 |
+(20 rows)
+
+-- omit "START" in DEFINE but it is ok because "START AS TRUE" is
+-- implicitly defined. per spec.
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value | nth_second
+----------+------------+-------+-------------+------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 140 | 07-02-2023
+ company1 | 07-02-2023 | 200 | | |
+ company1 | 07-03-2023 | 150 | | |
+ company1 | 07-04-2023 | 140 | | |
+ company1 | 07-05-2023 | 150 | | |
+ company1 | 07-06-2023 | 90 | 90 | 120 | 07-07-2023
+ company1 | 07-07-2023 | 110 | | |
+ company1 | 07-08-2023 | 130 | | |
+ company1 | 07-09-2023 | 120 | | |
+ company1 | 07-10-2023 | 130 | | |
+ company2 | 07-01-2023 | 50 | 50 | 1400 | 07-02-2023
+ company2 | 07-02-2023 | 2000 | | |
+ company2 | 07-03-2023 | 1500 | | |
+ company2 | 07-04-2023 | 1400 | | |
+ company2 | 07-05-2023 | 1500 | | |
+ company2 | 07-06-2023 | 60 | 60 | 1200 | 07-07-2023
+ company2 | 07-07-2023 | 1100 | | |
+ company2 | 07-08-2023 | 1300 | | |
+ company2 | 07-09-2023 | 1200 | | |
+ company2 | 07-10-2023 | 1300 | | |
+(20 rows)
+
+-- the first row start with less than or equal to 100
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (LOWPRICE UP+ DOWN+)
+ DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value
+----------+------------+-------+-------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 140
+ company1 | 07-02-2023 | 200 | |
+ company1 | 07-03-2023 | 150 | |
+ company1 | 07-04-2023 | 140 | |
+ company1 | 07-05-2023 | 150 | |
+ company1 | 07-06-2023 | 90 | 90 | 120
+ company1 | 07-07-2023 | 110 | |
+ company1 | 07-08-2023 | 130 | |
+ company1 | 07-09-2023 | 120 | |
+ company1 | 07-10-2023 | 130 | |
+ company2 | 07-01-2023 | 50 | 50 | 1400
+ company2 | 07-02-2023 | 2000 | |
+ company2 | 07-03-2023 | 1500 | |
+ company2 | 07-04-2023 | 1400 | |
+ company2 | 07-05-2023 | 1500 | |
+ company2 | 07-06-2023 | 60 | 60 | 1200
+ company2 | 07-07-2023 | 1100 | |
+ company2 | 07-08-2023 | 1300 | |
+ company2 | 07-09-2023 | 1200 | |
+ company2 | 07-10-2023 | 1300 | |
+(20 rows)
+
+-- second row raises 120%
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (LOWPRICE UP+ DOWN+)
+ DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price) * 1.2,
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value
+----------+------------+-------+-------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 140
+ company1 | 07-02-2023 | 200 | |
+ company1 | 07-03-2023 | 150 | |
+ company1 | 07-04-2023 | 140 | |
+ company1 | 07-05-2023 | 150 | |
+ company1 | 07-06-2023 | 90 | |
+ company1 | 07-07-2023 | 110 | |
+ company1 | 07-08-2023 | 130 | |
+ company1 | 07-09-2023 | 120 | |
+ company1 | 07-10-2023 | 130 | |
+ company2 | 07-01-2023 | 50 | 50 | 1400
+ company2 | 07-02-2023 | 2000 | |
+ company2 | 07-03-2023 | 1500 | |
+ company2 | 07-04-2023 | 1400 | |
+ company2 | 07-05-2023 | 1500 | |
+ company2 | 07-06-2023 | 60 | |
+ company2 | 07-07-2023 | 1100 | |
+ company2 | 07-08-2023 | 1300 | |
+ company2 | 07-09-2023 | 1200 | |
+ company2 | 07-10-2023 | 1300 | |
+(20 rows)
+
+-- using NEXT
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UPDOWN)
+ DEFINE
+ START AS TRUE,
+ UPDOWN AS price > PREV(price) AND price > NEXT(price)
+);
+ company | tdate | price | first_value | last_value
+----------+------------+-------+-------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 200
+ company1 | 07-02-2023 | 200 | |
+ company1 | 07-03-2023 | 150 | |
+ company1 | 07-04-2023 | 140 | 140 | 150
+ company1 | 07-05-2023 | 150 | |
+ company1 | 07-06-2023 | 90 | |
+ company1 | 07-07-2023 | 110 | 110 | 130
+ company1 | 07-08-2023 | 130 | |
+ company1 | 07-09-2023 | 120 | |
+ company1 | 07-10-2023 | 130 | |
+ company2 | 07-01-2023 | 50 | 50 | 2000
+ company2 | 07-02-2023 | 2000 | |
+ company2 | 07-03-2023 | 1500 | |
+ company2 | 07-04-2023 | 1400 | 1400 | 1500
+ company2 | 07-05-2023 | 1500 | |
+ company2 | 07-06-2023 | 60 | |
+ company2 | 07-07-2023 | 1100 | 1100 | 1300
+ company2 | 07-08-2023 | 1300 | |
+ company2 | 07-09-2023 | 1200 | |
+ company2 | 07-10-2023 | 1300 | |
+(20 rows)
+
+-- using AFTER MATCH SKIP TO NEXT ROW
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP TO NEXT ROW
+ INITIAL
+ PATTERN (START UPDOWN)
+ DEFINE
+ START AS TRUE,
+ UPDOWN AS price > PREV(price) AND price > NEXT(price)
+);
+ company | tdate | price | first_value | last_value
+----------+------------+-------+-------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 200
+ company1 | 07-02-2023 | 200 | |
+ company1 | 07-03-2023 | 150 | |
+ company1 | 07-04-2023 | 140 | 140 | 150
+ company1 | 07-05-2023 | 150 | |
+ company1 | 07-06-2023 | 90 | |
+ company1 | 07-07-2023 | 110 | 110 | 130
+ company1 | 07-08-2023 | 130 | |
+ company1 | 07-09-2023 | 120 | |
+ company1 | 07-10-2023 | 130 | |
+ company2 | 07-01-2023 | 50 | 50 | 2000
+ company2 | 07-02-2023 | 2000 | |
+ company2 | 07-03-2023 | 1500 | |
+ company2 | 07-04-2023 | 1400 | 1400 | 1500
+ company2 | 07-05-2023 | 1500 | |
+ company2 | 07-06-2023 | 60 | |
+ company2 | 07-07-2023 | 1100 | 1100 | 1300
+ company2 | 07-08-2023 | 1300 | |
+ company2 | 07-09-2023 | 1200 | |
+ company2 | 07-10-2023 | 1300 | |
+(20 rows)
+
+-- match everything
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ INITIAL
+ PATTERN (A+)
+ DEFINE
+ A AS TRUE
+);
+ company | tdate | price | first_value | last_value
+----------+------------+-------+-------------+------------
+ company1 | 07-01-2023 | 100 | 100 | 130
+ company1 | 07-02-2023 | 200 | |
+ company1 | 07-03-2023 | 150 | |
+ company1 | 07-04-2023 | 140 | |
+ company1 | 07-05-2023 | 150 | |
+ company1 | 07-06-2023 | 90 | |
+ company1 | 07-07-2023 | 110 | |
+ company1 | 07-08-2023 | 130 | |
+ company1 | 07-09-2023 | 120 | |
+ company1 | 07-10-2023 | 130 | |
+ company2 | 07-01-2023 | 50 | 50 | 1300
+ company2 | 07-02-2023 | 2000 | |
+ company2 | 07-03-2023 | 1500 | |
+ company2 | 07-04-2023 | 1400 | |
+ company2 | 07-05-2023 | 1500 | |
+ company2 | 07-06-2023 | 60 | |
+ company2 | 07-07-2023 | 1100 | |
+ company2 | 07-08-2023 | 1300 | |
+ company2 | 07-09-2023 | 1200 | |
+ company2 | 07-10-2023 | 1300 | |
+(20 rows)
+
+-- backtracking with reclassification of rows
+-- using AFTER MATCH SKIP PAST LAST ROW
+SELECT company, tdate, price, first_value(tdate) OVER w, last_value(tdate) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ INITIAL
+ PATTERN (A+ B+)
+ DEFINE
+ A AS price > 100,
+ B AS price > 100
+);
+ company | tdate | price | first_value | last_value
+----------+------------+-------+-------------+------------
+ company1 | 07-01-2023 | 100 | |
+ company1 | 07-02-2023 | 200 | 07-02-2023 | 07-05-2023
+ company1 | 07-03-2023 | 150 | |
+ company1 | 07-04-2023 | 140 | |
+ company1 | 07-05-2023 | 150 | |
+ company1 | 07-06-2023 | 90 | |
+ company1 | 07-07-2023 | 110 | 07-07-2023 | 07-10-2023
+ company1 | 07-08-2023 | 130 | |
+ company1 | 07-09-2023 | 120 | |
+ company1 | 07-10-2023 | 130 | |
+ company2 | 07-01-2023 | 50 | |
+ company2 | 07-02-2023 | 2000 | 07-02-2023 | 07-05-2023
+ company2 | 07-03-2023 | 1500 | |
+ company2 | 07-04-2023 | 1400 | |
+ company2 | 07-05-2023 | 1500 | |
+ company2 | 07-06-2023 | 60 | |
+ company2 | 07-07-2023 | 1100 | 07-07-2023 | 07-10-2023
+ company2 | 07-08-2023 | 1300 | |
+ company2 | 07-09-2023 | 1200 | |
+ company2 | 07-10-2023 | 1300 | |
+(20 rows)
+
+-- backtracking with reclassification of rows
+-- using AFTER MATCH SKIP TO NEXT ROW
+SELECT company, tdate, price, first_value(tdate) OVER w, last_value(tdate) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP TO NEXT ROW
+ INITIAL
+ PATTERN (A+ B+)
+ DEFINE
+ A AS price > 100,
+ B AS price > 100
+);
+ company | tdate | price | first_value | last_value
+----------+------------+-------+-------------+------------
+ company1 | 07-01-2023 | 100 | |
+ company1 | 07-02-2023 | 200 | 07-02-2023 | 07-05-2023
+ company1 | 07-03-2023 | 150 | 07-03-2023 | 07-05-2023
+ company1 | 07-04-2023 | 140 | 07-04-2023 | 07-05-2023
+ company1 | 07-05-2023 | 150 | |
+ company1 | 07-06-2023 | 90 | |
+ company1 | 07-07-2023 | 110 | 07-07-2023 | 07-10-2023
+ company1 | 07-08-2023 | 130 | 07-08-2023 | 07-10-2023
+ company1 | 07-09-2023 | 120 | 07-09-2023 | 07-10-2023
+ company1 | 07-10-2023 | 130 | |
+ company2 | 07-01-2023 | 50 | |
+ company2 | 07-02-2023 | 2000 | 07-02-2023 | 07-05-2023
+ company2 | 07-03-2023 | 1500 | 07-03-2023 | 07-05-2023
+ company2 | 07-04-2023 | 1400 | 07-04-2023 | 07-05-2023
+ company2 | 07-05-2023 | 1500 | |
+ company2 | 07-06-2023 | 60 | |
+ company2 | 07-07-2023 | 1100 | 07-07-2023 | 07-10-2023
+ company2 | 07-08-2023 | 1300 | 07-08-2023 | 07-10-2023
+ company2 | 07-09-2023 | 1200 | 07-09-2023 | 07-10-2023
+ company2 | 07-10-2023 | 1300 | |
+(20 rows)
+
+-- ROWS BETWEEN CURRENT ROW AND offset FOLLOWING
+SELECT company, tdate, price, first_value(tdate) OVER w, last_value(tdate) OVER w,
+ count(*) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value | count
+----------+------------+-------+-------------+------------+-------
+ company1 | 07-01-2023 | 100 | 07-01-2023 | 07-03-2023 | 3
+ company1 | 07-02-2023 | 200 | | | 0
+ company1 | 07-03-2023 | 150 | | | 0
+ company1 | 07-04-2023 | 140 | 07-04-2023 | 07-06-2023 | 3
+ company1 | 07-05-2023 | 150 | | | 0
+ company1 | 07-06-2023 | 90 | | | 0
+ company1 | 07-07-2023 | 110 | 07-07-2023 | 07-09-2023 | 3
+ company1 | 07-08-2023 | 130 | | | 0
+ company1 | 07-09-2023 | 120 | | | 0
+ company1 | 07-10-2023 | 130 | | | 0
+ company2 | 07-01-2023 | 50 | 07-01-2023 | 07-03-2023 | 3
+ company2 | 07-02-2023 | 2000 | | | 0
+ company2 | 07-03-2023 | 1500 | | | 0
+ company2 | 07-04-2023 | 1400 | 07-04-2023 | 07-06-2023 | 3
+ company2 | 07-05-2023 | 1500 | | | 0
+ company2 | 07-06-2023 | 60 | | | 0
+ company2 | 07-07-2023 | 1100 | 07-07-2023 | 07-09-2023 | 3
+ company2 | 07-08-2023 | 1300 | | | 0
+ company2 | 07-09-2023 | 1200 | | | 0
+ company2 | 07-10-2023 | 1300 | | | 0
+(20 rows)
+
+--
+-- Aggregates
+--
+-- using AFTER MATCH SKIP PAST LAST ROW
+SELECT company, tdate, price,
+ first_value(price) OVER w,
+ last_value(price) OVER w,
+ max(price) OVER w,
+ min(price) OVER w,
+ sum(price) OVER w,
+ avg(price) OVER w,
+ count(price) OVER w
+FROM stock
+WINDOW w AS (
+PARTITION BY company
+ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+AFTER MATCH SKIP PAST LAST ROW
+INITIAL
+PATTERN (START UP+ DOWN+)
+DEFINE
+START AS TRUE,
+UP AS price > PREV(price),
+DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value | max | min | sum | avg | count
+----------+------------+-------+-------------+------------+------+-----+------+-----------------------+-------
+ company1 | 07-01-2023 | 100 | 100 | 140 | 200 | 100 | 590 | 147.5000000000000000 | 4
+ company1 | 07-02-2023 | 200 | | | | | | | 0
+ company1 | 07-03-2023 | 150 | | | | | | | 0
+ company1 | 07-04-2023 | 140 | | | | | | | 0
+ company1 | 07-05-2023 | 150 | | | | | | | 0
+ company1 | 07-06-2023 | 90 | 90 | 120 | 130 | 90 | 450 | 112.5000000000000000 | 4
+ company1 | 07-07-2023 | 110 | | | | | | | 0
+ company1 | 07-08-2023 | 130 | | | | | | | 0
+ company1 | 07-09-2023 | 120 | | | | | | | 0
+ company1 | 07-10-2023 | 130 | | | | | | | 0
+ company2 | 07-01-2023 | 50 | 50 | 1400 | 2000 | 50 | 4950 | 1237.5000000000000000 | 4
+ company2 | 07-02-2023 | 2000 | | | | | | | 0
+ company2 | 07-03-2023 | 1500 | | | | | | | 0
+ company2 | 07-04-2023 | 1400 | | | | | | | 0
+ company2 | 07-05-2023 | 1500 | | | | | | | 0
+ company2 | 07-06-2023 | 60 | 60 | 1200 | 1300 | 60 | 3660 | 915.0000000000000000 | 4
+ company2 | 07-07-2023 | 1100 | | | | | | | 0
+ company2 | 07-08-2023 | 1300 | | | | | | | 0
+ company2 | 07-09-2023 | 1200 | | | | | | | 0
+ company2 | 07-10-2023 | 1300 | | | | | | | 0
+(20 rows)
+
+-- using AFTER MATCH SKIP TO NEXT ROW
+SELECT company, tdate, price,
+ first_value(price) OVER w,
+ last_value(price) OVER w,
+ max(price) OVER w,
+ min(price) OVER w,
+ sum(price) OVER w,
+ avg(price) OVER w,
+ count(price) OVER w
+FROM stock
+WINDOW w AS (
+PARTITION BY company
+ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+AFTER MATCH SKIP TO NEXT ROW
+INITIAL
+PATTERN (START UP+ DOWN+)
+DEFINE
+START AS TRUE,
+UP AS price > PREV(price),
+DOWN AS price < PREV(price)
+);
+ company | tdate | price | first_value | last_value | max | min | sum | avg | count
+----------+------------+-------+-------------+------------+------+------+------+-----------------------+-------
+ company1 | 07-01-2023 | 100 | 100 | 140 | 200 | 100 | 590 | 147.5000000000000000 | 4
+ company1 | 07-02-2023 | 200 | | | | | | | 0
+ company1 | 07-03-2023 | 150 | | | | | | | 0
+ company1 | 07-04-2023 | 140 | 140 | 90 | 150 | 90 | 380 | 126.6666666666666667 | 3
+ company1 | 07-05-2023 | 150 | | | | | | | 0
+ company1 | 07-06-2023 | 90 | 90 | 120 | 130 | 90 | 450 | 112.5000000000000000 | 4
+ company1 | 07-07-2023 | 110 | 110 | 120 | 130 | 110 | 360 | 120.0000000000000000 | 3
+ company1 | 07-08-2023 | 130 | | | | | | | 0
+ company1 | 07-09-2023 | 120 | | | | | | | 0
+ company1 | 07-10-2023 | 130 | | | | | | | 0
+ company2 | 07-01-2023 | 50 | 50 | 1400 | 2000 | 50 | 4950 | 1237.5000000000000000 | 4
+ company2 | 07-02-2023 | 2000 | | | | | | | 0
+ company2 | 07-03-2023 | 1500 | | | | | | | 0
+ company2 | 07-04-2023 | 1400 | 1400 | 60 | 1500 | 60 | 2960 | 986.6666666666666667 | 3
+ company2 | 07-05-2023 | 1500 | | | | | | | 0
+ company2 | 07-06-2023 | 60 | 60 | 1200 | 1300 | 60 | 3660 | 915.0000000000000000 | 4
+ company2 | 07-07-2023 | 1100 | 1100 | 1200 | 1300 | 1100 | 3600 | 1200.0000000000000000 | 3
+ company2 | 07-08-2023 | 1300 | | | | | | | 0
+ company2 | 07-09-2023 | 1200 | | | | | | | 0
+ company2 | 07-10-2023 | 1300 | | | | | | | 0
+(20 rows)
+
+-- JOIN case
+CREATE TEMP TABLE t1 (i int, v1 int);
+CREATE TEMP TABLE t2 (j int, v2 int);
+INSERT INTO t1 VALUES(1,10);
+INSERT INTO t1 VALUES(1,11);
+INSERT INTO t1 VALUES(1,12);
+INSERT INTO t2 VALUES(2,10);
+INSERT INTO t2 VALUES(2,11);
+INSERT INTO t2 VALUES(2,12);
+SELECT * FROM t1, t2 WHERE t1.v1 <= 11 AND t2.v2 <= 11;
+ i | v1 | j | v2
+---+----+---+----
+ 1 | 10 | 2 | 10
+ 1 | 10 | 2 | 11
+ 1 | 11 | 2 | 10
+ 1 | 11 | 2 | 11
+(4 rows)
+
+SELECT *, count(*) OVER w FROM t1, t2
+WINDOW w AS (
+ PARTITION BY t1.i
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (A)
+ DEFINE
+ A AS v1 <= 11 AND v2 <= 11
+);
+ i | v1 | j | v2 | count
+---+----+---+----+-------
+ 1 | 10 | 2 | 10 | 1
+ 1 | 10 | 2 | 11 | 1
+ 1 | 10 | 2 | 12 | 0
+ 1 | 11 | 2 | 10 | 1
+ 1 | 11 | 2 | 11 | 1
+ 1 | 11 | 2 | 12 | 0
+ 1 | 12 | 2 | 10 | 0
+ 1 | 12 | 2 | 11 | 0
+ 1 | 12 | 2 | 12 | 0
+(9 rows)
+
+-- WITH case
+WITH wstock AS (
+ SELECT * FROM stock WHERE tdate < '2023-07-08'
+)
+SELECT tdate, price,
+first_value(tdate) OVER w,
+count(*) OVER w
+ FROM wstock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ tdate | price | first_value | count
+------------+-------+-------------+-------
+ 07-01-2023 | 100 | 07-01-2023 | 4
+ 07-02-2023 | 200 | | 0
+ 07-03-2023 | 150 | | 0
+ 07-04-2023 | 140 | | 0
+ 07-05-2023 | 150 | | 0
+ 07-06-2023 | 90 | | 0
+ 07-07-2023 | 110 | | 0
+ 07-01-2023 | 50 | 07-01-2023 | 4
+ 07-02-2023 | 2000 | | 0
+ 07-03-2023 | 1500 | | 0
+ 07-04-2023 | 1400 | | 0
+ 07-05-2023 | 1500 | | 0
+ 07-06-2023 | 60 | | 0
+ 07-07-2023 | 1100 | | 0
+(14 rows)
+
+--
+-- Error cases
+--
+-- row pattern definition variable name must not appear more than once
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price),
+ UP AS price > PREV(price)
+);
+ERROR: row pattern definition variable name "up" appears more than once in DEFINE clause
+LINE 11: UP AS price > PREV(price),
+ ^
+-- subqueries in DEFINE clause are not supported
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START LOWPRICE)
+ DEFINE
+ START AS TRUE,
+ LOWPRICE AS price < (SELECT 100)
+);
+ERROR: cannot use subquery in DEFINE expression
+LINE 11: LOWPRICE AS price < (SELECT 100)
+ ^
+-- aggregates in DEFINE clause are not supported
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START LOWPRICE)
+ DEFINE
+ START AS TRUE,
+ LOWPRICE AS price < count(*)
+);
+ERROR: aggregate functions are not allowed in DEFINE
+LINE 11: LOWPRICE AS price < count(*)
+ ^
+-- FRAME must start at current row when row patttern recognition is used
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ERROR: FRAME must start at current row when row patttern recognition is used
+-- SEEK is not supported
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP TO NEXT ROW
+ SEEK
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+ERROR: SEEK is not supported
+LINE 8: SEEK
+ ^
+HINT: Use INITIAL.
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 969ced994f..3eaa36dca7 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -98,7 +98,7 @@ test: publication subscription
# Another group of parallel tests
# select_views depends on create_view
# ----------
-test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combocid tsearch tsdicts foreign_data window xmlmap functional_deps advisory_lock indirect_toast equivclass
+test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combocid tsearch tsdicts foreign_data window xmlmap functional_deps advisory_lock indirect_toast equivclass rpr
# ----------
# Another group of parallel tests (JSON related)
diff --git a/src/test/regress/sql/rpr.sql b/src/test/regress/sql/rpr.sql
new file mode 100644
index 0000000000..302e2b86a5
--- /dev/null
+++ b/src/test/regress/sql/rpr.sql
@@ -0,0 +1,405 @@
+--
+-- Test for row pattern definition clause
+--
+
+CREATE TEMP TABLE stock (
+ company TEXT,
+ tdate DATE,
+ price INTEGER
+);
+INSERT INTO stock VALUES ('company1', '2023-07-01', 100);
+INSERT INTO stock VALUES ('company1', '2023-07-02', 200);
+INSERT INTO stock VALUES ('company1', '2023-07-03', 150);
+INSERT INTO stock VALUES ('company1', '2023-07-04', 140);
+INSERT INTO stock VALUES ('company1', '2023-07-05', 150);
+INSERT INTO stock VALUES ('company1', '2023-07-06', 90);
+INSERT INTO stock VALUES ('company1', '2023-07-07', 110);
+INSERT INTO stock VALUES ('company1', '2023-07-08', 130);
+INSERT INTO stock VALUES ('company1', '2023-07-09', 120);
+INSERT INTO stock VALUES ('company1', '2023-07-10', 130);
+INSERT INTO stock VALUES ('company2', '2023-07-01', 50);
+INSERT INTO stock VALUES ('company2', '2023-07-02', 2000);
+INSERT INTO stock VALUES ('company2', '2023-07-03', 1500);
+INSERT INTO stock VALUES ('company2', '2023-07-04', 1400);
+INSERT INTO stock VALUES ('company2', '2023-07-05', 1500);
+INSERT INTO stock VALUES ('company2', '2023-07-06', 60);
+INSERT INTO stock VALUES ('company2', '2023-07-07', 1100);
+INSERT INTO stock VALUES ('company2', '2023-07-08', 1300);
+INSERT INTO stock VALUES ('company2', '2023-07-09', 1200);
+INSERT INTO stock VALUES ('company2', '2023-07-10', 1300);
+
+SELECT * FROM stock;
+
+-- basic test using PREV
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+-- basic test using PREV. UP appears twice
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+ UP+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+-- basic test using PREV. Use '*'
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP* DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+-- basic test with none greedy pattern
+SELECT company, tdate, price, count(*) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (A A A)
+ DEFINE
+ A AS price >= 140 AND price <= 150
+);
+
+-- last_value() should remain consistent
+SELECT company, tdate, price, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+-- omit "START" in DEFINE but it is ok because "START AS TRUE" is
+-- implicitly defined. per spec.
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
+ nth_value(tdate, 2) OVER w AS nth_second
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+-- the first row start with less than or equal to 100
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (LOWPRICE UP+ DOWN+)
+ DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+-- second row raises 120%
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (LOWPRICE UP+ DOWN+)
+ DEFINE
+ LOWPRICE AS price <= 100,
+ UP AS price > PREV(price) * 1.2,
+ DOWN AS price < PREV(price)
+);
+
+-- using NEXT
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UPDOWN)
+ DEFINE
+ START AS TRUE,
+ UPDOWN AS price > PREV(price) AND price > NEXT(price)
+);
+
+-- using AFTER MATCH SKIP TO NEXT ROW
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP TO NEXT ROW
+ INITIAL
+ PATTERN (START UPDOWN)
+ DEFINE
+ START AS TRUE,
+ UPDOWN AS price > PREV(price) AND price > NEXT(price)
+);
+
+-- match everything
+
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ INITIAL
+ PATTERN (A+)
+ DEFINE
+ A AS TRUE
+);
+
+-- backtracking with reclassification of rows
+-- using AFTER MATCH SKIP PAST LAST ROW
+SELECT company, tdate, price, first_value(tdate) OVER w, last_value(tdate) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ INITIAL
+ PATTERN (A+ B+)
+ DEFINE
+ A AS price > 100,
+ B AS price > 100
+);
+
+-- backtracking with reclassification of rows
+-- using AFTER MATCH SKIP TO NEXT ROW
+SELECT company, tdate, price, first_value(tdate) OVER w, last_value(tdate) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP TO NEXT ROW
+ INITIAL
+ PATTERN (A+ B+)
+ DEFINE
+ A AS price > 100,
+ B AS price > 100
+);
+
+-- ROWS BETWEEN CURRENT ROW AND offset FOLLOWING
+SELECT company, tdate, price, first_value(tdate) OVER w, last_value(tdate) OVER w,
+ count(*) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING
+ AFTER MATCH SKIP PAST LAST ROW
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+--
+-- Aggregates
+--
+
+-- using AFTER MATCH SKIP PAST LAST ROW
+SELECT company, tdate, price,
+ first_value(price) OVER w,
+ last_value(price) OVER w,
+ max(price) OVER w,
+ min(price) OVER w,
+ sum(price) OVER w,
+ avg(price) OVER w,
+ count(price) OVER w
+FROM stock
+WINDOW w AS (
+PARTITION BY company
+ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+AFTER MATCH SKIP PAST LAST ROW
+INITIAL
+PATTERN (START UP+ DOWN+)
+DEFINE
+START AS TRUE,
+UP AS price > PREV(price),
+DOWN AS price < PREV(price)
+);
+
+-- using AFTER MATCH SKIP TO NEXT ROW
+SELECT company, tdate, price,
+ first_value(price) OVER w,
+ last_value(price) OVER w,
+ max(price) OVER w,
+ min(price) OVER w,
+ sum(price) OVER w,
+ avg(price) OVER w,
+ count(price) OVER w
+FROM stock
+WINDOW w AS (
+PARTITION BY company
+ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+AFTER MATCH SKIP TO NEXT ROW
+INITIAL
+PATTERN (START UP+ DOWN+)
+DEFINE
+START AS TRUE,
+UP AS price > PREV(price),
+DOWN AS price < PREV(price)
+);
+
+-- JOIN case
+CREATE TEMP TABLE t1 (i int, v1 int);
+CREATE TEMP TABLE t2 (j int, v2 int);
+INSERT INTO t1 VALUES(1,10);
+INSERT INTO t1 VALUES(1,11);
+INSERT INTO t1 VALUES(1,12);
+INSERT INTO t2 VALUES(2,10);
+INSERT INTO t2 VALUES(2,11);
+INSERT INTO t2 VALUES(2,12);
+
+SELECT * FROM t1, t2 WHERE t1.v1 <= 11 AND t2.v2 <= 11;
+
+SELECT *, count(*) OVER w FROM t1, t2
+WINDOW w AS (
+ PARTITION BY t1.i
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (A)
+ DEFINE
+ A AS v1 <= 11 AND v2 <= 11
+);
+
+-- WITH case
+WITH wstock AS (
+ SELECT * FROM stock WHERE tdate < '2023-07-08'
+)
+SELECT tdate, price,
+first_value(tdate) OVER w,
+count(*) OVER w
+ FROM wstock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+--
+-- Error cases
+--
+
+-- row pattern definition variable name must not appear more than once
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price),
+ UP AS price > PREV(price)
+);
+
+-- subqueries in DEFINE clause are not supported
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START LOWPRICE)
+ DEFINE
+ START AS TRUE,
+ LOWPRICE AS price < (SELECT 100)
+);
+
+-- aggregates in DEFINE clause are not supported
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START LOWPRICE)
+ DEFINE
+ START AS TRUE,
+ LOWPRICE AS price < count(*)
+);
+
+-- FRAME must start at current row when row patttern recognition is used
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
+ INITIAL
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
+
+-- SEEK is not supported
+SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
+ FROM stock
+ WINDOW w AS (
+ PARTITION BY company
+ ORDER BY tdate
+ ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+ AFTER MATCH SKIP TO NEXT ROW
+ SEEK
+ PATTERN (START UP+ DOWN+)
+ DEFINE
+ START AS TRUE,
+ UP AS price > PREV(price),
+ DOWN AS price < PREV(price)
+);
--
2.25.1
----Next_Part(Fri_May_24_11_39_19_2024_763)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v20-0008-Allow-to-print-raw-parse-tree.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v7 4/5] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--C2xzVbxFmVrP7k6O
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v9 4/5] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--aq/6bi8L6WORnI+9
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v10 4/5] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index aecc7a127e6..edf5ac65660 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--NVvBxFuyV/R+1/8/
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0005-Change-RELATION-and-DATABASE-stats-to-anytime-fl.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v1 2/3] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
3 files changed, 2 insertions(+), 57 deletions(-)
75.3% src/backend/replication/
24.6% src/backend/utils/activity/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index a41453530a1..266379c780a 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -553,16 +553,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1ab09655a70..c33185bd337 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1826,7 +1822,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1847,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1958,8 +1952,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1968,15 +1961,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2879,8 +2863,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2975,9 +2957,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2985,7 +2964,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2996,21 +2974,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
--
2.34.1
--tjEWjIIwfNIHQLgt
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v1-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v4 3/4] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.9% src/backend/replication/
22.8% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 6970af3f3ff..dcbe3517b46 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -565,16 +565,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a0e6a3d200c..74102def9c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1825,7 +1821,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1846,7 +1841,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1957,8 +1951,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1967,15 +1960,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2878,8 +2862,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2974,9 +2956,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2984,7 +2963,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2995,21 +2973,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index d7e62e4d488..dda872f7074 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--wR/mWGXukst4NraF
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v11 4/5] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index aecc7a127e6..edf5ac65660 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--2MEBAGW8+kohXisi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0005-Change-RELATION-and-DATABASE-stats-to-anytime-fl.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v8 4/5] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--OI5irqZsxWxBW9nT
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v3 2/3] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
3 files changed, 2 insertions(+), 57 deletions(-)
75.3% src/backend/replication/
24.6% src/backend/utils/activity/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index a41453530a1..266379c780a 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -553,16 +553,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1ab09655a70..c33185bd337 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1826,7 +1822,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1847,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1958,8 +1952,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1968,15 +1961,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2879,8 +2863,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2975,9 +2957,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2985,7 +2964,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2996,21 +2974,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
--
2.34.1
--lCc1MWfC7i2uqs8g
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v6 4/5] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.4% src/test/recovery/t/
3.5% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..bceec2adede 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--DxqFthphbM+ha9Dy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v11 4/5] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index aecc7a127e6..edf5ac65660 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--2MEBAGW8+kohXisi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0005-Change-RELATION-and-DATABASE-stats-to-anytime-fl.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v2 2/3] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
3 files changed, 2 insertions(+), 57 deletions(-)
75.3% src/backend/replication/
24.6% src/backend/utils/activity/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index a41453530a1..266379c780a 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -553,16 +553,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1ab09655a70..c33185bd337 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1826,7 +1822,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1847,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1958,8 +1952,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1968,15 +1961,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2879,8 +2863,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2975,9 +2957,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2985,7 +2964,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2996,21 +2974,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
--
2.34.1
--OFsrIl+bjhifp5hk
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v5 3/4] Remove useless calls to flush some stats
@ 2026-01-06 11:06 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Bertrand Drouvot @ 2026-01-06 11:06 UTC (permalink / raw)
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.9% src/backend/replication/
22.8% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 24d7ef795cb..1c5ffcab3e0 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -572,16 +572,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a0e6a3d200c..74102def9c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1825,7 +1821,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1846,7 +1841,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1957,8 +1951,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1967,15 +1960,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2878,8 +2862,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2974,9 +2956,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2984,7 +2963,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2995,21 +2973,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index d7e62e4d488..dda872f7074 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
--WLtTrIUEfQDmERxy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* Add pg_stat_vfdcache view for VFD cache statistics
@ 2026-03-21 16:58 KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
0 siblings, 1 reply; 28+ messages in thread
From: KAZAR Ayoub @ 2026-03-21 16:58 UTC (permalink / raw)
To: pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
Hello hackers,
This comes from Tomas's patch idea from his website[1], i thought this
patch makes sense to have.
PostgreSQL's virtual file descriptor (VFD) maintains a
per-backend cache of open file descriptors, bounded by
max_files_per_process (default 1000). When the cache is full, the
least-recently-used entry is evicted so its OS fd is closed, so a new
file can be opened. On the next access to that file, open() must be
called again, incurring a syscall that a larger cache would have
avoided.
A trivial example is with partitioned tables: a table with 1500
partitions requires even more than 1500 file descriptors per full scan (main
fork, vm ...), which is more than the default limit, causing potential
evictions and reopens.
The problem is well-understood and the fix is straightforward: raise
max_files_per_process. Tomas showed a 4-5x throughput
improvement in [1] sometimes, on my end i see something less than that,
depending on the query itself, but we get the idea.
AFAIK there is currently no way from inside PostgreSQL to know whether fd
cache pressure is occurring.
Implementation is trivial, because the VFD cache is strictly per-backend,
the counters are also
per-backend and require no shared memory or locking. Three macros
(pgstat_count_vfd_hit/miss/eviction) update fields in PendingVfdCacheStats
directly from fd.c.
I find this a bit useful, I would love to hear about anyone's thoughts
whether this is useful or not.
[1] https://vondra.me/posts/patch-idea-statistics-for-file-descriptor-cache
Regards,
Ayoub
Attachments:
[text/x-patch] 0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch (21.8K, ../../CA+K2RumP33Cpj--88E+rNADa8fzSBBiav=rvzyaMM=sYNaOkfA@mail.gmail.com/3-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch)
download | inline diff:
From 1124d345945ecd68ca6019e6f1d24f1a5cb81861 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Sat, 21 Mar 2026 17:03:10 +0100
Subject: [PATCH] Add pg_stat_vfdcache view for VFD cache statistics
PostgreSQL's virtual file descriptor (VFD) layer maintains a
per-backend cache of open file descriptors bounded by
max_files_per_process (default 1000). When the cache is full, the
least-recently-used entry is evicted (its OS fd closed) so a new file
can be opened. A subsequent access to an evicted file must call
open() again.
A trivial example is with partitioned tables: a table with 1500
partitions requires up to many file descriptors per full scan (main
fork, vm ...), which is more than the default limit, causing
potential evictions and reopens.
To my knowledge there's no way to observe this from inside PostgreSQL.
This commit adds:
pg_stat_vfdcache -- a view exposing, for the current session:
hits number of FileAccess() calls that found the fd open
misses number of FileAccess() calls that found fd closed
evictions number of LRU evictions (close() to free a slot)
cache_size physical fds currently open
max_cache_size current value of max_files_per_process
hit_ratio hits / (hits + misses)
stats_reset timestamp of last counter reset
pg_stat_reset_vfdcache() -- resets counters for the current session
Because the VFD cache is strictly per-backend, the counters are
backend-local and require no shared memory, locking, or atomic
operations. Three macros (pgstat_count_vfd_hit/miss/eviction)
update fields in PendingVfdCacheStats directly from fd.c.
Hit and miss counters are placed in FileAccess(), which is the
single gate through which all VFD-mediated file reads, writes,
truncations, and size checks pass. The eviction counter is placed
in ReleaseLruFile(), before LruDelete() is called.
---
doc/src/sgml/monitoring.sgml | 138 +++++++++++++++++++
src/backend/catalog/system_views.sql | 17 +++
src/backend/storage/file/fd.c | 22 +++
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat_vfdcache.c | 75 ++++++++++
src/backend/utils/adt/pgstatfuncs.c | 52 +++++++
src/include/catalog/pg_proc.dat | 36 +++++
src/include/pgstat.h | 31 +++++
src/include/storage/fd.h | 1 +
src/test/regress/expected/rules.out | 10 ++
11 files changed, 384 insertions(+)
create mode 100644 src/backend/utils/activity/pgstat_vfdcache.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 462019a972c..23ea643fcab 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -472,6 +472,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structname>pg_stat_vfdcache</structname>
+ <indexterm><primary>pg_stat_vfdcache</primary></indexterm>
+ </para>
+ <para>
+ One row only, showing statistics about the current session's virtual
+ file descriptor (VFD) cache activity. See
+ <link linkend="monitoring-pg-stat-vfdcache-view">
+ <structname>pg_stat_vfdcache</structname></link> for details.
+ </para></entry>
+ </row>
+
<row>
<entry><structname>pg_stat_checkpointer</structname><indexterm><primary>pg_stat_checkpointer</primary></indexterm></entry>
<entry>One row only, showing statistics about the
@@ -3352,6 +3365,118 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</sect2>
+ <sect2 id="monitoring-pg-stat-vfdcache-view">
+ <title><structname>pg_stat_vfdcache</structname></title>
+
+ <indexterm zone="monitoring-pg-stat-vfdcache-view">
+ <primary>pg_stat_vfdcache</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_vfdcache</structname> view will always have a
+ single row, containing data about the VFD (Virtual File Descriptor) cache activity of the
+ current session.
+ </para>
+
+ <table id="pg-stat-vfdcache-view" xreflabel="pg_stat_vfdcache">
+ <title><structname>pg_stat_vfdcache</structname> View</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">Column</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hits</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of file accesses where the physical file descriptor was
+ already open in the cache, requiring no system call.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>misses</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of file accesses where the physical file descriptor had
+ been evicted from the cache, requiring <function>open()</function>
+ to be called again before the access could proceed.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of times a physical file descriptor was closed to make
+ room in the cache for a new one (LRU eviction). Each eviction
+ will eventually produce a miss when the evicted file is accessed
+ again. When the cache is thrashing, this value will be close to
+ or equal to <structfield>misses</structfield>.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_size</structfield> <type>integer</type>
+ </para></entry>
+ <entry><type>integer</type></entry>
+ <entry>
+ Current number of physical file descriptors open in the VFD
+ cache at the time the view is queried.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>max_cache_size</structfield> <type>integer</type>
+ </para></entry>
+ <entry><type>integer</type></entry>
+ <entry>
+ Current value of <varname>max_files_per_process</varname>. The
+ effective limit on simultaneously open file descriptors may be
+ lower than this value if the operating system's per-process file
+ descriptor limit is smaller.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hit_ratio</structfield> <type>float8</type>
+ </para></entry>
+ <entry><type>float8</type></entry>
+ <entry>
+ Fraction of file accesses that were cache hits:
+ <literal>hits / (hits + misses)</literal>. A value of
+ <literal>1.0</literal> indicates no cache pressure. Values
+ significantly below <literal>1.0</literal> indicate that
+ <varname>max_files_per_process</varname> should be increased.
+ <literal>NULL</literal> if no accesses have been recorded yet.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield>
+ <type>timestamp with time zone</type>
+ </para></entry>
+ <entry><type>timestamp with time zone</type></entry>
+ <entry>
+ Time at which the counters were last reset by
+ <function>pg_stat_reset_vfdcache()</function>.
+ <literal>NULL</literal> if the counters have never been reset.
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-pg-stat-checkpointer-view">
<title><structname>pg_stat_checkpointer</structname></title>
@@ -5530,6 +5655,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm><primary>pg_stat_reset_vfdcache</primary></indexterm>
+ <function>pg_stat_reset_vfdcache</function> ()
+ <returnvalue>void</returnvalue>
+ </para>
+ <para>
+ Reset the VFD cache statistics counters for the current session to
+ zero. The reset timestamp is recorded in
+ <structname>pg_stat_vfdcache</structname>.<structfield>stats_reset</structfield>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f1ed7b58f13..e395fe556eb 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1507,3 +1507,20 @@ CREATE VIEW pg_aios AS
SELECT * FROM pg_get_aios();
REVOKE ALL ON pg_aios FROM PUBLIC;
GRANT SELECT ON pg_aios TO pg_read_all_stats;
+
+CREATE VIEW pg_stat_vfdcache AS
+ SELECT
+ pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_evictions() AS evictions,
+ pg_stat_get_vfd_cache_size() AS cache_size,
+ current_setting('max_files_per_process')::int AS max_cache_size,
+ CASE
+ WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
+ THEN NULL::float8
+ ELSE pg_stat_get_vfd_hits()::float8
+ / (pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses())
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
+
+GRANT SELECT ON pg_stat_vfdcache TO PUBLIC;
\ No newline at end of file
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 01f1bd6e687..480ad604115 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1378,6 +1378,7 @@ ReleaseLruFile(void)
* in the ring.
*/
Assert(VfdCache[0].lruMoreRecently != 0);
+ pgstat_count_vfd_eviction();
LruDelete(VfdCache[0].lruMoreRecently);
return true; /* freed a file */
}
@@ -1491,6 +1492,7 @@ FileAccess(File file)
if (FileIsNotOpen(file))
{
+ pgstat_count_vfd_miss();
returnValue = LruInsert(file);
if (returnValue != 0)
return returnValue;
@@ -1501,14 +1503,34 @@ FileAccess(File file)
* We now know that the file is open and that it is not the last one
* accessed, so we need to move it to the head of the Lru ring.
*/
+ pgstat_count_vfd_hit();
Delete(file);
Insert(file);
}
+ else
+ {
+ /* fd is open and already at MRU end */
+ pgstat_count_vfd_hit();
+ }
return 0;
}
+/*
+ * GetVfdCacheOccupancy
+ *
+ * Return the number of physical file descriptors currently open in the
+ * VFD cache (nfile). This is the live cache size exposed by
+ * pg_stat_vfdcache.cache_size.
+ *
+ */
+int
+GetVfdCacheOccupancy(void)
+{
+ return nfile;
+}
+
/*
* Called whenever a temporary file is deleted to report its size.
*/
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index c37bfb350bb..da9ee955aec 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -31,6 +31,7 @@ OBJS = \
pgstat_shmem.o \
pgstat_slru.o \
pgstat_subscription.o \
+ pgstat_vfdcache.o \
pgstat_wal.o \
pgstat_xact.o \
wait_event.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 53bd5a246ca..0178d446106 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -16,6 +16,7 @@ backend_sources += files(
'pgstat_shmem.c',
'pgstat_slru.c',
'pgstat_subscription.c',
+ 'pgstat_vfdcache.c',
'pgstat_wal.c',
'pgstat_xact.c',
)
diff --git a/src/backend/utils/activity/pgstat_vfdcache.c b/src/backend/utils/activity/pgstat_vfdcache.c
new file mode 100644
index 00000000000..ca117b09ee6
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_vfdcache.c
@@ -0,0 +1,75 @@
+/*-------------------------------------------------------------------------
+ *
+ * pgstat_vfdcache.c
+ * Implementation of VFD (Virtual File Descriptor) cache statistics.
+ *
+ * The VFD cache in fd.c maintains a cache of open file
+ * descriptors, bounded by max_files_per_process. When the cache is full,
+ * the least-recently-used entry is evicted (its OS fd closed) so a new
+ * file can be opened. A subsequent access to an evicted VFD must call
+ * open() again, incurring a syscall that a warm cache would have avoided.
+ *
+ * This module tracks hits (fd was open, no syscall), misses (fd was
+ * closed, open() required), and evictions (close() to make room) for the
+ * current backend. Because the VFD cache is strictly per-backend, these
+ * counters are also per-backend so no shared memory or locking is used.
+ *
+ * The view pg_stat_vfdcache exposes these counters for the current session
+ * together with the live cache occupancy and the configured limit, giving
+ * possibility to diagnose fd-cache pressure.
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_vfdcache.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "pgstat.h"
+#include "storage/fd.h"
+#include "utils/timestamp.h"
+
+/*
+ * Per-backend VFD cache counters.
+ *
+ * Updated directly by fd.c via the inline macros in pgstat.h. Because
+ * only the owning backend ever writes these variables, no atomic ops or
+ * locks are required.
+ */
+PgStat_VfdCacheStats PendingVfdCacheStats;
+
+/*
+ * Timestamp of the last pg_stat_reset_vfdcache() call for this backend.
+ * Zero just means "never reset".
+ */
+static TimestampTz vfd_stats_reset_timestamp = 0;
+
+/*
+ * pgstat_fetch_stat_vfdcache
+ *
+ * Return a pointer to a filled PgStat_VfdCacheStats for the current
+ * backend. The returned pointer is valid until the next call.
+ */
+PgStat_VfdCacheStats *
+pgstat_fetch_stat_vfdcache(void)
+{
+ /* counters live directly in PendingVfdCacheStats; just attach timestamp */
+ PendingVfdCacheStats.stat_reset_timestamp = vfd_stats_reset_timestamp;
+ return &PendingVfdCacheStats;
+}
+
+/*
+ * pgstat_reset_vfdcache
+ *
+ * Reset all VFD cache counters for the current backend and record the
+ * reset timestamp.
+ */
+void
+pgstat_reset_vfdcache(void)
+{
+ memset(&PendingVfdCacheStats, 0, sizeof(PendingVfdCacheStats));
+ vfd_stats_reset_timestamp = GetCurrentTimestamp();
+}
\ No newline at end of file
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5f907335990..24b8e0208c8 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1340,6 +1340,58 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+Datum
+pg_stat_get_vfd_hits(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_hits);
+}
+
+Datum
+pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
+}
+
+Datum
+pg_stat_get_vfd_evictions(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_evictions);
+}
+
+Datum
+pg_stat_get_vfd_cache_size(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT32(GetVfdCacheOccupancy());
+}
+
+/*
+ * pg_stat_get_vfd_stat_reset_time
+ * Timestamp of the last pg_stat_reset_vfdcache() call, or NULL if
+ * the counters have never been reset.
+ */
+Datum
+pg_stat_get_vfd_stat_reset_time(PG_FUNCTION_ARGS)
+{
+ TimestampTz ts = pgstat_fetch_stat_vfdcache()->stat_reset_timestamp;
+
+ if (ts == 0)
+ PG_RETURN_NULL();
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+}
+
+/*
+ * pg_stat_reset_vfdcache
+ * Reset VFD cache counters for the current backend.
+ */
+Datum
+pg_stat_reset_vfdcache(PG_FUNCTION_ARGS)
+{
+ pgstat_reset_vfdcache();
+ PG_RETURN_VOID();
+}
+
+
/*
* When adding a new column to the pg_stat_io view and the
* pg_stat_get_backend_io() function, add a new enum value here above
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 84e7adde0e5..ac46f7e9c25 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12842,5 +12842,41 @@
{ oid => '8281', descr => 'hash',
proname => 'hashoid8extended', prorettype => 'int8',
proargtypes => 'oid8 int8', prosrc => 'hashoid8extended' },
+{ oid => '9560',
+ descr => 'statistics: number of VFD cache hits in current session',
+ proname => 'pg_stat_get_vfd_hits',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_hits' },
+{ oid => '9561',
+ descr => 'statistics: number of VFD cache misses in current session',
+ proname => 'pg_stat_get_vfd_misses',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_misses' },
+{ oid => '9562',
+ descr => 'statistics: number of VFD cache evictions in current session',
+ proname => 'pg_stat_get_vfd_evictions',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_evictions' },
+{ oid => '9563',
+ descr => 'statistics: number of physical fds currently open in VFD cache',
+ proname => 'pg_stat_get_vfd_cache_size',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int4', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_size' },
+{ oid => '9564',
+ descr => 'statistics: timestamp of last VFD cache stats reset',
+ proname => 'pg_stat_get_vfd_stat_reset_time',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'timestamptz', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_stat_reset_time' },
+{ oid => '9565',
+ descr => 'statistics: reset VFD cache counters for the current session',
+ proname => 'pg_stat_reset_vfdcache',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'void', proargtypes => '',
+ prosrc => 'pg_stat_reset_vfdcache' },
]
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 216b93492ba..ffbc1584a70 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -271,6 +271,22 @@ typedef struct PgStat_CheckpointerStats
TimestampTz stat_reset_timestamp;
} PgStat_CheckpointerStats;
+/* ---------
+ * PgStat_VfdCacheStats Virtual File Descriptor cache statistics
+ *
+ * Tracks hit/miss/eviction events in the per-backend VFD cache (fd.c).
+ * Because the VFD cache is strictly per-backend these counters are updated
+ * without any locking. The view pg_stat_vfdcache exposes them together
+ * with the current cache occupancy.
+ * ---------
+ */
+typedef struct PgStat_VfdCacheStats
+{
+ PgStat_Counter vfd_hits; /* fd was open, no open() syscall needed */
+ PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was required */
+ PgStat_Counter vfd_evictions; /* close() called to free a slot for a new fd */
+ TimestampTz stat_reset_timestamp;
+} PgStat_VfdCacheStats;
/*
* Types related to counting IO operations
@@ -592,6 +608,15 @@ extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void);
extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern PgStat_VfdCacheStats *pgstat_fetch_stat_vfdcache(void);
+extern void pgstat_reset_vfdcache(void);
+#define pgstat_count_vfd_hit() (PendingVfdCacheStats.vfd_hits++)
+#define pgstat_count_vfd_miss() (PendingVfdCacheStats.vfd_misses++)
+#define pgstat_count_vfd_eviction() (PendingVfdCacheStats.vfd_evictions++)
/*
* Functions in pgstat_io.c
@@ -823,6 +848,12 @@ extern PGDLLIMPORT int pgstat_fetch_consistency;
/* updated directly by bgwriter and bufmgr */
extern PGDLLIMPORT PgStat_BgWriterStats PendingBgWriterStats;
+/*
+ * Variables in pgstat_vfdcache.c
+ */
+
+/* updated directly by fd.cn (per-backend) */
+extern PGDLLIMPORT PgStat_VfdCacheStats PendingVfdCacheStats;
/*
* Variables in pgstat_checkpointer.c
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 8ac466fd346..9a54bbdb18a 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -149,6 +149,7 @@ extern char *FilePathName(File file);
extern int FileGetRawDesc(File file);
extern int FileGetRawFlags(File file);
extern mode_t FileGetRawMode(File file);
+extern int GetVfdCacheOccupancy(void);
/* Operations used for sharing named temporary files */
extern File PathNameCreateTemporaryFile(const char *path, bool error_on_failure);
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 32bea58db2c..be717049da1 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2356,6 +2356,16 @@ pg_stat_user_tables| SELECT relid,
stats_reset
FROM pg_stat_all_tables
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
+pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_evictions() AS evictions,
+ pg_stat_get_vfd_cache_size() AS cache_size,
+ (current_setting('max_files_per_process'::text))::integer AS max_cache_size,
+ CASE
+ WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
+ ELSE ((pg_stat_get_vfd_hits())::double precision / ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()))::double precision)
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
--
2.34.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-03-23 11:22 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
0 siblings, 1 reply; 28+ messages in thread
From: Jakub Wartak @ 2026-03-23 11:22 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
On Sat, Mar 21, 2026 at 5:59 PM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> Hello hackers,
>
> This comes from Tomas's patch idea from his website[1], i thought this patch makes sense to have.
>
> PostgreSQL's virtual file descriptor (VFD) maintains a
> per-backend cache of open file descriptors, bounded by
> max_files_per_process (default 1000). When the cache is full, the
> least-recently-used entry is evicted so its OS fd is closed, so a new
> file can be opened. On the next access to that file, open() must be
> called again, incurring a syscall that a larger cache would have
> avoided.
>
> A trivial example is with partitioned tables: a table with 1500
> partitions requires even more than 1500 file descriptors per full scan (main
> fork, vm ...), which is more than the default limit, causing potential evictions and reopens.
>
> The problem is well-understood and the fix is straightforward: raise
> max_files_per_process. Tomas showed a 4-5x throughput
> improvement in [1] sometimes, on my end i see something less than that, depending on the query itself, but we get the idea.
>
> AFAIK there is currently no way from inside PostgreSQL to know whether fd cache pressure is occurring.
>
> Implementation is trivial, because the VFD cache is strictly per-backend, the counters are also
> per-backend and require no shared memory or locking. Three macros (pgstat_count_vfd_hit/miss/eviction) update fields in PendingVfdCacheStats directly from fd.c.
>
> I find this a bit useful, I would love to hear about anyone's thoughts whether this is useful or not.
Hi,
My $0.02, for that for that to being useful it would need to allow viewing
global vfd cache picture (across all backends), not just from *current* backend.
Applicaiton wouldn't call this function anyway, because they would have to be
modified.
In order to get that you technically should collect the hits/misses in local
pending pgstat io area (see e.g. pgstat_io or simpler pgstat_bgwriter/
checkpointer) like you do already with PendingVfdCacheStats, but then copy them
to shared memory pgstat area (with some LWLock* protection) that would be
queryable.
-J.
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
@ 2026-03-23 12:35 ` David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 28+ messages in thread
From: David Geier @ 2026-03-23 12:35 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
Hi!
On 23.03.2026 12:22, Jakub Wartak wrote:
> On Sat, Mar 21, 2026 at 5:59 PM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>>
>> Hello hackers,
>>
>> This comes from Tomas's patch idea from his website[1], i thought this patch makes sense to have.
>>
>> PostgreSQL's virtual file descriptor (VFD) maintains a
>> per-backend cache of open file descriptors, bounded by
>> max_files_per_process (default 1000). When the cache is full, the
>> least-recently-used entry is evicted so its OS fd is closed, so a new
>> file can be opened. On the next access to that file, open() must be
>> called again, incurring a syscall that a larger cache would have
>> avoided.
That's one use-case. The other one that I've recently come across is
just knowing how many VFD cache entries there are in the first place.
While the number of open files is bounded by max_files_per_process, the
number of cache entries is unbounded. Large database can easily have
hundreds of thousands of files due to our segmentation scheme.
Workloads that access a big portion of these files can end up spending
very considerable amounts of memory on the VFD cache. For example, with
100,000 VFD entries per backend * 80 bytes per VFD = ~7.6 MiB. With 1000
backends that almost 10 GiB just for VFD entries; assuming that each
backend over time accumulates that many files.
A production database I looked recently had ~300,000 files and many
thousand backends. It spent close to 30 GiBs on VFD cache.
I've looked at struct vfd and some simple changes to the struct would
already cut memory consumption in half. I can look into that.
Thoughts?
>> A trivial example is with partitioned tables: a table with 1500
>> partitions requires even more than 1500 file descriptors per full scan (main
>> fork, vm ...), which is more than the default limit, causing potential evictions and reopens.
>>
>> The problem is well-understood and the fix is straightforward: raise
>> max_files_per_process. Tomas showed a 4-5x throughput
>> improvement in [1] sometimes, on my end i see something less than that, depending on the query itself, but we get the idea.
The question is what the kernel makes out of that, especially in
aforementioned case where the number of total files and backends is large.
In the Linux kernel each process that open some file gets its own struct
file. sizeof(struct file) is ~200 bytes. Hence, increasing
max_files_per_process can measurably impact memory consumption if
changed lightheartedly. We should document that.
But I guess in most cases it's rather about changing it from 1k to 2k,
rather than changing it from 1k to 100k.
>> AFAIK there is currently no way from inside PostgreSQL to know whether fd cache pressure is occurring.
>>
>> Implementation is trivial, because the VFD cache is strictly per-backend, the counters are also
>> per-backend and require no shared memory or locking. Three macros (pgstat_count_vfd_hit/miss/eviction) update fields in PendingVfdCacheStats directly from fd.c.
>>
>> I find this a bit useful, I would love to hear about anyone's thoughts whether this is useful or not.
>
> Hi,
>
> My $0.02, for that for that to being useful it would need to allow viewing
> global vfd cache picture (across all backends), not just from *current* backend.
> Applicaiton wouldn't call this function anyway, because they would have to be
> modified.
+1
> In order to get that you technically should collect the hits/misses in local
> pending pgstat io area (see e.g. pgstat_io or simpler pgstat_bgwriter/
> checkpointer) like you do already with PendingVfdCacheStats, but then copy them
> to shared memory pgstat area (with some LWLock* protection) that would be
> queryable.
I would include here the sum of VFD cache entries across all backend and
the total VFD cache size.
--
David Geier
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
@ 2026-03-23 23:36 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
0 siblings, 1 reply; 28+ messages in thread
From: KAZAR Ayoub @ 2026-03-23 23:36 UTC (permalink / raw)
To: David Geier <geidav.pg@gmail.com>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
Hello David and Jakub,
On Mon, Mar 23, 2026 at 1:35 PM David Geier <geidav.pg@gmail.com> wrote:
> Hi!
>
> On 23.03.2026 12:22, Jakub Wartak wrote:
> > On Sat, Mar 21, 2026 at 5:59 PM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >>
> >> Hello hackers,
> >>
> >> This comes from Tomas's patch idea from his website[1], i thought this
> patch makes sense to have.
> >>
> >> PostgreSQL's virtual file descriptor (VFD) maintains a
> >> per-backend cache of open file descriptors, bounded by
> >> max_files_per_process (default 1000). When the cache is full, the
> >> least-recently-used entry is evicted so its OS fd is closed, so a new
> >> file can be opened. On the next access to that file, open() must be
> >> called again, incurring a syscall that a larger cache would have
> >> avoided.
>
> That's one use-case. The other one that I've recently come across is
> just knowing how many VFD cache entries there are in the first place.
>
> While the number of open files is bounded by max_files_per_process, the
> number of cache entries is unbounded. Large database can easily have
> hundreds of thousands of files due to our segmentation scheme.
>
> Workloads that access a big portion of these files can end up spending
> very considerable amounts of memory on the VFD cache. For example, with
> 100,000 VFD entries per backend * 80 bytes per VFD = ~7.6 MiB. With 1000
> backends that almost 10 GiB just for VFD entries; assuming that each
> backend over time accumulates that many files.
>
> A production database I looked recently had ~300,000 files and many
> thousand backends. It spent close to 30 GiBs on VFD cache.
>
> I've looked at struct vfd and some simple changes to the struct would
> already cut memory consumption in half. I can look into that.
>
> Thoughts?
Looking forward to this.
What also bothers me in that space is if a backend allocates 100K entries
in VFD cache, that cache is never shrank ever again,
the cache only grows (if it needs more than its lifetime maximum) until the
backend dies, although this is useful as entries are reused if free instead
of
allocating entries, whether a spike in files openings effects a long living
backend to keep holding a useless amount of
cache size it will need in the future, i don't imagine this to be common
though, what do you think about this issue from your experience ?
>
> In the Linux kernel each process that open some file gets its own struct
> file. sizeof(struct file) is ~200 bytes. Hence, increasing
> max_files_per_process can measurably impact memory consumption if
> changed lightheartedly. We should document that.
>
> But I guess in most cases it's rather about changing it from 1k to 2k,
> rather than changing it from 1k to 100k.
>
Indeed, I agree with this.
>
> >> AFAIK there is currently no way from inside PostgreSQL to know whether
> fd cache pressure is occurring.
> >>
> >> Implementation is trivial, because the VFD cache is strictly
> per-backend, the counters are also
> >> per-backend and require no shared memory or locking. Three macros
> (pgstat_count_vfd_hit/miss/eviction) update fields in PendingVfdCacheStats
> directly from fd.c.
> >>
> >> I find this a bit useful, I would love to hear about anyone's thoughts
> whether this is useful or not.
> >
> > Hi,
> >
> > My $0.02, for that for that to being useful it would need to allow
> viewing
> > global vfd cache picture (across all backends), not just from *current*
> backend.
> > Applicaiton wouldn't call this function anyway, because they would have
> to be
> > modified.
>
> +1
>
Would it be reasonable to have both ? I changed the way i was thinking
about it,
Its clear one would want global vfd cache behavior monitoring as its
configuration is global but its effect is mostly backend specific,
That leads me to think it could also be useful to ALSO maintain per-backend
metrics, to help identify which backends are going crazy with the VFD cache.
>
> > In order to get that you technically should collect the hits/misses in
> local
> > pending pgstat io area (see e.g. pgstat_io or simpler pgstat_bgwriter/
> > checkpointer) like you do already with PendingVfdCacheStats, but then
> copy them
> > to shared memory pgstat area (with some LWLock* protection) that would be
> > queryable.
>
> I would include here the sum of VFD cache entries across all backend and
> the total VFD cache size.
>
I'll be doing this soon.
>
> --
> David Geier
>
Regards,
Ayoub Kazar
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-03-24 14:09 ` David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 28+ messages in thread
From: David Geier @ 2026-03-24 14:09 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
On 24.03.2026 00:36, KAZAR Ayoub wrote:
>> I've looked at struct vfd and some simple changes to the struct would
>> already cut memory consumption in half. I can look into that.
>>
>> Thoughts?
>
> Looking forward to this.
I try to come up with something the next days.
> What also bothers me in that space is if a backend allocates 100K entries
> in VFD cache, that cache is never shrank ever again,
> the cache only grows (if it needs more than its lifetime maximum) until the
> backend dies, although this is useful as entries are reused if free instead
> of
> allocating entries, whether a spike in files openings effects a long living
> backend to keep holding a useless amount of
> cache size it will need in the future, i don't imagine this to be common
> though, what do you think about this issue from your experience ?
Currently the cache is directly mapped by the VFD index. That means we
could only resize down to the maximum used VFD index.
Being able to resize independently of the maximum VFD index would
require changing to a hash map like simplehash.h. I can take a look how
invasive such a change would be.
--
David Geier
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
@ 2026-03-29 18:23 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 28+ messages in thread
From: KAZAR Ayoub @ 2026-03-29 18:23 UTC (permalink / raw)
To: David Geier <geidav.pg@gmail.com>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
Hello,
On Tue, Mar 24, 2026 at 3:09 PM David Geier <geidav.pg@gmail.com> wrote:
> On 24.03.2026 00:36, KAZAR Ayoub wrote:
>
> >> I've looked at struct vfd and some simple changes to the struct would
> >> already cut memory consumption in half. I can look into that.
> >>
> >> Thoughts?
> >
> > Looking forward to this.
>
> I try to come up with something the next days.
>
> > What also bothers me in that space is if a backend allocates 100K entries
> > in VFD cache, that cache is never shrank ever again,
> > the cache only grows (if it needs more than its lifetime maximum) until
> the
> > backend dies, although this is useful as entries are reused if free
> instead
> > of
> > allocating entries, whether a spike in files openings effects a long
> living
> > backend to keep holding a useless amount of
> > cache size it will need in the future, i don't imagine this to be common
> > though, what do you think about this issue from your experience ?
>
> Currently the cache is directly mapped by the VFD index. That means we
> could only resize down to the maximum used VFD index.
>
> Being able to resize independently of the maximum VFD index would
> require changing to a hash map like simplehash.h. I can take a look how
> invasive such a change would be.
> --
> David Geier
>
I've implemented the recommended global stats view on vfd cache, the
implementation should be also straightforward as it follows the same
cumulative shared statistics infrastructure like pgstat_bgwriter and others
do.
Attached is v2 patch also contains what David suggested for global cache
size and entries in the view.
Kind regards,
Ayoub
Attachments:
[text/x-patch] v2-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch (33.6K, ../../CA+K2RuncGCW55b-XUe8gJGwdMSOgTFKG-uSCgbVDZ+HewiT5EQ@mail.gmail.com/3-v2-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch)
download | inline diff:
From fccfca3156a49428e58f1ff0dc66b3ca24e73730 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Sat, 21 Mar 2026 17:03:10 +0100
Subject: [PATCH] Add pg_stat_vfdcache view for VFD cache statistics
PostgreSQL's virtual file descriptor (VFD) layer maintains a
per-backend cache of open file descriptors bounded by
max_files_per_process (default 1000). When the cache is full, the
least-recently-used entry is evicted (its OS fd closed) so a new file
can be opened. A subsequent access to an evicted file must call
open() again.
A trivial example is with partitioned tables: a table with 1500
partitions requires up to many file descriptors per full scan (main
fork, vm ...), which is more than the default limit, causing
potential evictions and reopens.
This commit adds:
pg_stat_vfdcache -- a single-row view exposing cluster-wide VFD cache
statistics:
hits number of VFD cache hits
misses number of VFD cache misses
evictions number of LRU evictions
cache_entries total allocated VFD entries across backends
cache_bytes total VFD entry memory footprint (bytes)
max_files_per_process current value of max_files_per_process
hit_ratio hits / (hits + misses)
stats_reset timestamp of last counter reset
pg_stat_reset_vfdcache() -- resets shared VFD counters
The implementation follows the same cumulative shared statistics infrastructure like pgstat_bgwriter and others do.
Event counting remains cheap in backend-local pending storage and is flushed
into shared fixed stats. In addition, each backend publishes VFD gauge values
(vfd_entries, vfd_cache_bytes) into backend stats, and SQL accessors sum these
per-backend gauges at read time to produce cluster totals.
Hit and miss counters are placed in FileAccess(), which is the
single gate through which all VFD-mediated file reads, writes,
truncations, and size checks pass. The eviction counter is placed
in ReleaseLruFile(), before LruDelete() is called.
---
doc/src/sgml/monitoring.sgml | 148 ++++++++++++++++++
src/backend/catalog/system_views.sql | 18 +++
src/backend/storage/file/fd.c | 49 ++++++
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 17 ++
src/backend/utils/activity/pgstat_vfdcache.c | 154 +++++++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 123 ++++++++++++++-
src/include/catalog/pg_proc.dat | 42 +++++
src/include/pgstat.h | 45 ++++++
src/include/storage/fd.h | 3 +
src/include/utils/pgstat_internal.h | 22 +++
src/include/utils/pgstat_kind.h | 3 +-
src/test/regress/expected/rules.out | 11 ++
src/test/regress/expected/stats.out | 16 +-
src/test/regress/sql/stats.sql | 5 +
16 files changed, 655 insertions(+), 3 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_vfdcache.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 462019a972c..15543762a7e 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -472,6 +472,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structname>pg_stat_vfdcache</structname>
+ <indexterm><primary>pg_stat_vfdcache</primary></indexterm>
+ </para>
+ <para>
+ One row only, showing cluster-wide statistics about virtual file
+ descriptor (VFD) cache activity. See
+ <link linkend="monitoring-pg-stat-vfdcache-view">
+ <structname>pg_stat_vfdcache</structname></link> for details.
+ </para></entry>
+ </row>
+
<row>
<entry><structname>pg_stat_checkpointer</structname><indexterm><primary>pg_stat_checkpointer</primary></indexterm></entry>
<entry>One row only, showing statistics about the
@@ -3352,6 +3365,128 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</sect2>
+ <sect2 id="monitoring-pg-stat-vfdcache-view">
+ <title><structname>pg_stat_vfdcache</structname></title>
+
+ <indexterm zone="monitoring-pg-stat-vfdcache-view">
+ <primary>pg_stat_vfdcache</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_vfdcache</structname> view will always have a
+ single row, containing data about cluster-wide VFD (Virtual File
+ Descriptor) cache activity.
+ </para>
+
+ <table id="pg-stat-vfdcache-view" xreflabel="pg_stat_vfdcache">
+ <title><structname>pg_stat_vfdcache</structname> View</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">Column</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hits</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of file accesses where the physical file descriptor was
+ already open in the cache, requiring no system call.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>misses</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of file accesses where the physical file descriptor had
+ been evicted from the cache, requiring <function>open()</function>
+ to be called again before the access could proceed.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of times a physical file descriptor was closed to make
+ room in the cache for a new one (LRU eviction). Each eviction
+ will eventually produce a miss when the evicted file is accessed
+ again. When the cache is thrashing, this value will be close to
+ or equal to <structfield>misses</structfield>.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_entries</structfield> <type>integer</type>
+ </para></entry>
+ <entry><type>integer</type></entry>
+ <entry>
+ Sum of currently allocated VFD cache entries across all active
+ backends.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_bytes</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Sum of memory used by allocated VFD cache entries across all active
+ backends, in bytes.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>max_files_per_process</structfield> <type>integer</type>
+ </para></entry>
+ <entry><type>integer</type></entry>
+ <entry>
+ Current value of <varname>max_files_per_process</varname>. The
+ effective limit on simultaneously open file descriptors may be
+ lower than this value if the operating system's per-process file
+ descriptor limit is smaller.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hit_ratio</structfield> <type>float8</type>
+ </para></entry>
+ <entry><type>float8</type></entry>
+ <entry>
+ Fraction of file accesses that were cache hits:
+ <literal>hits / (hits + misses)</literal>. A value of
+ <literal>1.0</literal> indicates no cache pressure. Values
+ significantly below <literal>1.0</literal> indicate that
+ <varname>max_files_per_process</varname> should be increased.
+ <literal>NULL</literal> if no accesses have been recorded yet.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield>
+ <type>timestamp with time zone</type>
+ </para></entry>
+ <entry><type>timestamp with time zone</type></entry>
+ <entry>
+ Time at which the counters were last reset by
+ <function>pg_stat_reset_vfdcache()</function>.
+ <literal>NULL</literal> if the counters have never been reset.
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-pg-stat-checkpointer-view">
<title><structname>pg_stat_checkpointer</structname></title>
@@ -5530,6 +5665,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm><primary>pg_stat_reset_vfdcache</primary></indexterm>
+ <function>pg_stat_reset_vfdcache</function> ()
+ <returnvalue>void</returnvalue>
+ </para>
+ <para>
+ Reset shared VFD cache statistics counters to zero. The reset
+ timestamp is recorded in
+ <structname>pg_stat_vfdcache</structname>.<structfield>stats_reset</structfield>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f1ed7b58f13..67e7a37abe8 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1507,3 +1507,21 @@ CREATE VIEW pg_aios AS
SELECT * FROM pg_get_aios();
REVOKE ALL ON pg_aios FROM PUBLIC;
GRANT SELECT ON pg_aios TO pg_read_all_stats;
+
+CREATE VIEW pg_stat_vfdcache AS
+ SELECT
+ pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_evictions() AS evictions,
+ pg_stat_get_vfd_cache_size() AS cache_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
+ current_setting('max_files_per_process')::int AS max_files_per_process,
+ CASE
+ WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
+ THEN NULL::float8
+ ELSE pg_stat_get_vfd_hits()::float8
+ / (pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses())
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
+
+GRANT SELECT ON pg_stat_vfdcache TO PUBLIC;
\ No newline at end of file
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 01f1bd6e687..b273c173b11 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1378,6 +1378,7 @@ ReleaseLruFile(void)
* in the ring.
*/
Assert(VfdCache[0].lruMoreRecently != 0);
+ pgstat_count_vfd_eviction();
LruDelete(VfdCache[0].lruMoreRecently);
return true; /* freed a file */
}
@@ -1491,6 +1492,7 @@ FileAccess(File file)
if (FileIsNotOpen(file))
{
+ pgstat_count_vfd_miss();
returnValue = LruInsert(file);
if (returnValue != 0)
return returnValue;
@@ -1501,14 +1503,61 @@ FileAccess(File file)
* We now know that the file is open and that it is not the last one
* accessed, so we need to move it to the head of the Lru ring.
*/
+ pgstat_count_vfd_hit();
Delete(file);
Insert(file);
}
+ else
+ {
+ /* fd is open and already at MRU end */
+ pgstat_count_vfd_hit();
+ }
return 0;
}
+/*
+ * GetVfdCacheOccupancy
+ *
+ * Return the number of physical file descriptors currently open in the
+ * VFD cache (nfile). This is the live cache size exposed by
+ * pg_stat_vfdcache.cache_size.
+ *
+ */
+int
+GetVfdCacheOccupancy(void)
+{
+ return nfile;
+}
+
+/*
+ * GetVfdCacheEntries
+ *
+ * Return the number of VFD cache entries currently allocated for this
+ * backend, excluding slot 0 which is the freelist/LRU header.
+ */
+uint64
+GetVfdCacheEntries(void)
+{
+ if (SizeVfdCache == 0)
+ return 0;
+
+ return (uint64) (SizeVfdCache - 1);
+}
+
+/*
+ * GetVfdCacheBytes
+ *
+ * Return the memory footprint in bytes of currently allocated per-backend
+ * VFD entries, excluding slot 0.
+ */
+uint64
+GetVfdCacheBytes(void)
+{
+ return GetVfdCacheEntries() * sizeof(Vfd);
+}
+
/*
* Called whenever a temporary file is deleted to report its size.
*/
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index c37bfb350bb..da9ee955aec 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -31,6 +31,7 @@ OBJS = \
pgstat_shmem.o \
pgstat_slru.o \
pgstat_subscription.o \
+ pgstat_vfdcache.o \
pgstat_wal.o \
pgstat_xact.o \
wait_event.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 53bd5a246ca..0178d446106 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -16,6 +16,7 @@ backend_sources += files(
'pgstat_shmem.c',
'pgstat_slru.c',
'pgstat_subscription.c',
+ 'pgstat_vfdcache.c',
'pgstat_wal.c',
'pgstat_xact.c',
)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..85bf9748cc9 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -482,6 +482,23 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
+
+ [PGSTAT_KIND_VFDCACHE] = {
+ .name = "vfdcache",
+
+ .fixed_amount = true,
+ .write_to_file = true,
+
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, vfdcache),
+ .shared_ctl_off = offsetof(PgStat_ShmemControl, vfdcache),
+ .shared_data_off = offsetof(PgStatShared_VfdCache, stats),
+ .shared_data_len = sizeof(((PgStatShared_VfdCache *) 0)->stats),
+
+ .flush_static_cb = pgstat_vfdcache_flush_cb,
+ .init_shmem_cb = pgstat_vfdcache_init_shmem_cb,
+ .reset_all_cb = pgstat_vfdcache_reset_all_cb,
+ .snapshot_cb = pgstat_vfdcache_snapshot_cb,
+ },
};
/*
diff --git a/src/backend/utils/activity/pgstat_vfdcache.c b/src/backend/utils/activity/pgstat_vfdcache.c
new file mode 100644
index 00000000000..069b833e66e
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_vfdcache.c
@@ -0,0 +1,154 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_vfdcache.c
+ * Implementation of VFD cache statistics.
+ *
+ * VFD events are first counted in backend-local pending storage and then
+ * flushed into shared-memory cumulative stats, following the same model as
+ * other fixed stats kinds.
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_vfdcache.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "pgstat.h"
+#include "storage/fd.h"
+#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+
+/*
+ * Backend-local VFD counters waiting to be flushed.
+ */
+PgStat_VfdCacheStats PendingVfdCacheStats = {0};
+
+/*
+ * Count a VFD cache hit.
+ */
+void
+pgstat_count_vfd_hit(void)
+{
+ PendingVfdCacheStats.vfd_hits++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Count a VFD cache miss.
+ */
+void
+pgstat_count_vfd_miss(void)
+{
+ PendingVfdCacheStats.vfd_misses++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Count a VFD cache eviction.
+ */
+void
+pgstat_count_vfd_eviction(void)
+{
+ PendingVfdCacheStats.vfd_evictions++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Flush out backend-local pending VFD cache stats.
+ */
+bool
+pgstat_vfdcache_flush_cb(bool nowait)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_Backend *shbackendent;
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ if (pg_memory_is_all_zeros(&PendingVfdCacheStats,
+ sizeof(struct PgStat_VfdCacheStats)))
+ return false;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
+ MyProcNumber, nowait);
+ if (!entry_ref)
+ return true;
+
+ shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
+
+ shbackendent->stats.vfd_stats.vfd_entries = (PgStat_Counter) GetVfdCacheEntries();
+ shbackendent->stats.vfd_stats.vfd_cache_bytes = (PgStat_Counter) GetVfdCacheBytes();
+
+ pgstat_unlock_entry(entry_ref);
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.vfd_hits += PendingVfdCacheStats.vfd_hits;
+ stats_shmem->stats.vfd_misses += PendingVfdCacheStats.vfd_misses;
+ stats_shmem->stats.vfd_evictions += PendingVfdCacheStats.vfd_evictions;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ MemSet(&PendingVfdCacheStats, 0, sizeof(PendingVfdCacheStats));
+
+ return false;
+}
+
+/*
+ * Support function for SQL-callable pg_stat_get_vfd_* functions.
+ */
+PgStat_VfdCacheStats *
+pgstat_fetch_stat_vfdcache(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_VFDCACHE);
+
+ return &pgStatLocal.snapshot.vfdcache;
+}
+
+void
+pgstat_vfdcache_init_shmem_cb(void *stats)
+{
+ PgStatShared_VfdCache *stats_shmem = (PgStatShared_VfdCache *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+void
+pgstat_vfdcache_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_vfdcache_snapshot_cb(void)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+ PgStat_VfdCacheStats *reset_offset = &stats_shmem->reset_offset;
+ PgStat_VfdCacheStats reset;
+
+ pgstat_copy_changecounted_stats(&pgStatLocal.snapshot.vfdcache,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ pgStatLocal.snapshot.vfdcache.vfd_hits -= reset.vfd_hits;
+ pgStatLocal.snapshot.vfdcache.vfd_misses -= reset.vfd_misses;
+ pgStatLocal.snapshot.vfdcache.vfd_evictions -= reset.vfd_evictions;
+}
+
+void
+pgstat_reset_vfdcache(void)
+{
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5f907335990..ef9025e7f3f 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "access/xact.h"
#include "access/htup_details.h"
#include "access/xlog.h"
#include "access/xlogprefetcher.h"
@@ -28,6 +29,7 @@
#include "replication/logicallauncher.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/fd.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/timestamp.h"
@@ -1340,6 +1342,122 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+ * Sum per-backend VFD gauges across currently active backends.
+ */
+static void
+pgstat_get_vfd_backend_sums(PgStat_Counter *entries_sum,
+ PgStat_Counter *bytes_sum)
+{
+ static TimestampTz cached_stmt_start_ts = 0;
+ static PgStat_Counter cached_entries_sum = 0;
+ static PgStat_Counter cached_bytes_sum = 0;
+ TimestampTz stmt_start_ts = GetCurrentStatementStartTimestamp();
+ int num_backends = pgstat_fetch_stat_numbackends();
+
+ if (cached_stmt_start_ts == stmt_start_ts)
+ {
+ *entries_sum = cached_entries_sum;
+ *bytes_sum = cached_bytes_sum;
+ return;
+ }
+
+ *entries_sum = 0;
+ *bytes_sum = 0;
+
+ for (int curr_backend = 1; curr_backend <= num_backends; curr_backend++)
+ {
+ LocalPgBackendStatus *local_beentry;
+ PgBackendStatus *beentry;
+ PgStat_Backend *backend_stats;
+
+ local_beentry = pgstat_get_local_beentry_by_index(curr_backend);
+ beentry = &local_beentry->backendStatus;
+
+ if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
+ continue;
+
+ backend_stats = pgstat_fetch_stat_backend(local_beentry->proc_number);
+ if (!backend_stats)
+ continue;
+
+ *entries_sum += backend_stats->vfd_stats.vfd_entries;
+ *bytes_sum += backend_stats->vfd_stats.vfd_cache_bytes;
+ }
+
+ cached_entries_sum = *entries_sum;
+ cached_bytes_sum = *bytes_sum;
+ cached_stmt_start_ts = stmt_start_ts;
+}
+
+Datum
+pg_stat_get_vfd_hits(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_hits);
+}
+
+Datum
+pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
+}
+
+Datum
+pg_stat_get_vfd_evictions(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_evictions);
+}
+
+Datum
+pg_stat_get_vfd_cache_size(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter entries_sum;
+ PgStat_Counter bytes_dummy;
+
+ pgstat_get_vfd_backend_sums(&entries_sum, &bytes_dummy);
+
+ PG_RETURN_INT32((int32) entries_sum);
+}
+
+Datum
+pg_stat_get_vfd_cache_bytes(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter entries_sum;
+ PgStat_Counter bytes_sum;
+
+ pgstat_get_vfd_backend_sums(&entries_sum, &bytes_sum);
+
+ PG_RETURN_INT64(bytes_sum);
+}
+
+/*
+ * pg_stat_get_vfd_stat_reset_time
+ * Timestamp of the last pg_stat_reset_vfdcache() call, or NULL if
+ * the counters have never been reset.
+ */
+Datum
+pg_stat_get_vfd_stat_reset_time(PG_FUNCTION_ARGS)
+{
+ TimestampTz ts = pgstat_fetch_stat_vfdcache()->stat_reset_timestamp;
+
+ if (ts == 0)
+ PG_RETURN_NULL();
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+}
+
+/*
+ * pg_stat_reset_vfdcache
+ * Reset shared VFD cache counters.
+ */
+Datum
+pg_stat_reset_vfdcache(PG_FUNCTION_ARGS)
+{
+ pgstat_reset_vfdcache();
+ PG_RETURN_VOID();
+}
+
+
/*
* When adding a new column to the pg_stat_io view and the
* pg_stat_get_backend_io() function, add a new enum value here above
@@ -1928,6 +2046,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
PG_RETURN_VOID();
}
@@ -1948,11 +2067,13 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
else if (strcmp(target, "wal") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
+ else if (strcmp(target, "vfdcache") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", \"vfdcache\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 84e7adde0e5..d3bf2690a01 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12842,5 +12842,47 @@
{ oid => '8281', descr => 'hash',
proname => 'hashoid8extended', prorettype => 'int8',
proargtypes => 'oid8 int8', prosrc => 'hashoid8extended' },
+{ oid => '9560',
+ descr => 'statistics: number of VFD cache hits',
+ proname => 'pg_stat_get_vfd_hits',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_hits' },
+{ oid => '9561',
+ descr => 'statistics: number of VFD cache misses',
+ proname => 'pg_stat_get_vfd_misses',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_misses' },
+{ oid => '9562',
+ descr => 'statistics: number of VFD cache evictions',
+ proname => 'pg_stat_get_vfd_evictions',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_evictions' },
+{ oid => '9563',
+ descr => 'statistics: total number of allocated VFD cache entries',
+ proname => 'pg_stat_get_vfd_cache_size',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int4', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_size' },
+{ oid => '9566',
+ descr => 'statistics: total memory footprint of VFD cache entries in bytes',
+ proname => 'pg_stat_get_vfd_cache_bytes',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_bytes' },
+{ oid => '9564',
+ descr => 'statistics: timestamp of last VFD cache stats reset',
+ proname => 'pg_stat_get_vfd_stat_reset_time',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'timestamptz', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_stat_reset_time' },
+{ oid => '9565',
+ descr => 'statistics: reset shared VFD cache counters',
+ proname => 'pg_stat_reset_vfdcache',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'void', proargtypes => '',
+ prosrc => 'pg_stat_reset_vfdcache' },
]
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 216b93492ba..d8ddde2935d 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -271,6 +271,35 @@ typedef struct PgStat_CheckpointerStats
TimestampTz stat_reset_timestamp;
} PgStat_CheckpointerStats;
+/* ---------
+ * PgStat_VfdCacheStats Virtual File Descriptor cache statistics
+ *
+ * Tracks hit/miss/eviction events in the VFD cache (fd.c). These counters
+ * are accumulated in shared fixed stats and exposed by pg_stat_vfdcache.
+ * ---------
+ */
+typedef struct PgStat_VfdCacheStats
+{
+ PgStat_Counter vfd_hits; /* fd was open, no open() syscall needed */
+ PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was required */
+ PgStat_Counter vfd_evictions; /* close() called to free a slot for a new
+ * fd */
+ TimestampTz stat_reset_timestamp;
+} PgStat_VfdCacheStats;
+
+/* ---------
+ * PgStat_BackendVfdCacheStats VFD cache stats stored per backend
+ *
+ * Keeps per-backend VFD gauges in PGSTAT_KIND_BACKEND entries.
+ * vfd_entries and vfd_cache_bytes represent the current backend-local VFD
+ * cache footprint and are used to derive cluster-wide totals.
+ * ---------
+ */
+typedef struct PgStat_BackendVfdCacheStats
+{
+ PgStat_Counter vfd_entries;
+ PgStat_Counter vfd_cache_bytes;
+} PgStat_BackendVfdCacheStats;
/*
* Types related to counting IO operations
@@ -502,6 +531,7 @@ typedef struct PgStat_WalStats
typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
+ PgStat_BackendVfdCacheStats vfd_stats;
PgStat_BktypeIO io_stats;
PgStat_WalCounters wal_counters;
} PgStat_Backend;
@@ -592,6 +622,15 @@ extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void);
extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern PgStat_VfdCacheStats * pgstat_fetch_stat_vfdcache(void);
+extern void pgstat_reset_vfdcache(void);
+extern void pgstat_count_vfd_hit(void);
+extern void pgstat_count_vfd_miss(void);
+extern void pgstat_count_vfd_eviction(void);
/*
* Functions in pgstat_io.c
@@ -823,6 +862,12 @@ extern PGDLLIMPORT int pgstat_fetch_consistency;
/* updated directly by bgwriter and bufmgr */
extern PGDLLIMPORT PgStat_BgWriterStats PendingBgWriterStats;
+/*
+ * Variables in pgstat_vfdcache.c
+ */
+
+/* updated by VFD counting functions called from fd.c */
+extern PGDLLIMPORT PgStat_VfdCacheStats PendingVfdCacheStats;
/*
* Variables in pgstat_checkpointer.c
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 8ac466fd346..e666a59f18d 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -149,6 +149,9 @@ extern char *FilePathName(File file);
extern int FileGetRawDesc(File file);
extern int FileGetRawFlags(File file);
extern mode_t FileGetRawMode(File file);
+extern int GetVfdCacheOccupancy(void);
+extern uint64 GetVfdCacheEntries(void);
+extern uint64 GetVfdCacheBytes(void);
/* Operations used for sharing named temporary files */
extern File PathNameCreateTemporaryFile(const char *path, bool error_on_failure);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..4c55243304a 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -478,6 +478,15 @@ typedef struct PgStatShared_Wal
PgStat_WalStats stats;
} PgStatShared_Wal;
+typedef struct PgStatShared_VfdCache
+{
+ /* lock protects ->reset_offset as well as stats->stat_reset_timestamp */
+ LWLock lock;
+ uint32 changecount;
+ PgStat_VfdCacheStats stats;
+ PgStat_VfdCacheStats reset_offset;
+} PgStatShared_VfdCache;
+
/* ----------
@@ -572,6 +581,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+ PgStatShared_VfdCache vfdcache;
/*
* Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
@@ -606,6 +616,8 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ PgStat_VfdCacheStats vfdcache;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -719,6 +731,16 @@ extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern bool pgstat_vfdcache_flush_cb(bool nowait);
+extern void pgstat_vfdcache_init_shmem_cb(void *stats);
+extern void pgstat_vfdcache_reset_all_cb(TimestampTz ts);
+extern void pgstat_vfdcache_snapshot_cb(void);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index c30b6235623..4197f8c3589 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -38,9 +38,10 @@
#define PGSTAT_KIND_IO 10
#define PGSTAT_KIND_SLRU 11
#define PGSTAT_KIND_WAL 12
+#define PGSTAT_KIND_VFDCACHE 13
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
+#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_VFDCACHE
#define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1)
/* Custom stats kinds */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 32bea58db2c..402348125ea 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2356,6 +2356,17 @@ pg_stat_user_tables| SELECT relid,
stats_reset
FROM pg_stat_all_tables
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
+pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_evictions() AS evictions,
+ pg_stat_get_vfd_cache_size() AS cache_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
+ (current_setting('max_files_per_process'::text))::integer AS max_files_per_process,
+ CASE
+ WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
+ ELSE ((pg_stat_get_vfd_hits())::double precision / ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()))::double precision)
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index b99462bf946..6513f13747e 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1127,10 +1127,24 @@ SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
t
(1 row)
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+ ?column?
+----------
+ t
+(1 row)
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
ERROR: unrecognized reset target: "unknown"
-HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", or "wal".
+HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", "vfdcache", or "wal".
-- Test that reset works for pg_stat_database and pg_stat_database_conflicts
-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so that we
-- have a baseline for comparison. The same for pg_stat_database_conflicts as it shares
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 941222cf0be..42d8ffb4a2f 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -520,6 +520,11 @@ SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset
SELECT pg_stat_reset_shared('wal');
SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
--
2.34.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-03-29 19:46 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-31 14:54 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-31 18:27 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Tomas Vondra <tomas@vondra.me>
0 siblings, 2 replies; 28+ messages in thread
From: KAZAR Ayoub @ 2026-03-29 19:46 UTC (permalink / raw)
To: David Geier <geidav.pg@gmail.com>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
Rebased v2.
On Sun, Mar 29, 2026 at 8:23 PM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
> On Tue, Mar 24, 2026 at 3:09 PM David Geier <geidav.pg@gmail.com> wrote:
>
>> On 24.03.2026 00:36, KAZAR Ayoub wrote:
>>
>> >> I've looked at struct vfd and some simple changes to the struct would
>> >> already cut memory consumption in half. I can look into that.
>> >>
>> >> Thoughts?
>> >
>> > Looking forward to this.
>>
>> I try to come up with something the next days.
>>
>> > What also bothers me in that space is if a backend allocates 100K
>> entries
>> > in VFD cache, that cache is never shrank ever again,
>> > the cache only grows (if it needs more than its lifetime maximum) until
>> the
>> > backend dies, although this is useful as entries are reused if free
>> instead
>> > of
>> > allocating entries, whether a spike in files openings effects a long
>> living
>> > backend to keep holding a useless amount of
>> > cache size it will need in the future, i don't imagine this to be common
>> > though, what do you think about this issue from your experience ?
>>
>> Currently the cache is directly mapped by the VFD index. That means we
>> could only resize down to the maximum used VFD index.
>>
>> Being able to resize independently of the maximum VFD index would
>> require changing to a hash map like simplehash.h. I can take a look how
>> invasive such a change would be.
>
>
>> --
>> David Geier
>>
> I've implemented the recommended global stats view on vfd cache, the
> implementation should be also straightforward as it follows the same
> cumulative shared statistics infrastructure like pgstat_bgwriter and others
> do.
>
> Attached is v2 patch also contains what David suggested for global cache
> size and entries in the view.
>
> Kind regards,
> Ayoub
>
Attachments:
[text/x-patch] v2-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch (33.6K, ../../CA+K2RumDZ05pru3-YSZxe3Y==vO0hsahiweyvJw6QsjuGR5WsA@mail.gmail.com/3-v2-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch)
download | inline diff:
From 32261b18df3a49a159a7702f59a4c6bb6d8808f9 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Sun, 29 Mar 2026 21:40:23 +0200
Subject: [PATCH] Add pg_stat_vfdcache view for VFD cache statistics
PostgreSQL's virtual file descriptor (VFD) layer maintains a
per-backend cache of open file descriptors bounded by
max_files_per_process (default 1000). When the cache is full, the
least-recently-used entry is evicted (its OS fd closed) so a new file
can be opened. A subsequent access to an evicted file must call
open() again.
A trivial example is with partitioned tables: a table with 1500
partitions requires up to many file descriptors per full scan (main
fork, vm ...), which is more than the default limit, causing
potential evictions and reopens.
This commit adds:
pg_stat_vfdcache -- a single-row view exposing cluster-wide VFD cache
statistics:
hits number of VFD cache hits
misses number of VFD cache misses
evictions number of LRU evictions
cache_entries total allocated VFD entries across backends
cache_bytes total VFD entry memory footprint (bytes)
max_files_per_process current value of max_files_per_process
hit_ratio hits / (hits + misses)
stats_reset timestamp of last counter reset
pg_stat_reset_vfdcache() -- resets shared VFD counters
The implementation follows the same cumulative shared statistics infrastructure like pgstat_bgwriter and others do.
Event counting remains cheap in backend-local pending storage and is flushed
into shared fixed stats. In addition, each backend publishes VFD gauge values
(vfd_entries, vfd_cache_bytes) into backend stats, and SQL accessors sum these
per-backend gauges at read time to produce cluster totals.
Hit and miss counters are placed in FileAccess(), which is the
single gate through which all VFD-mediated file reads, writes,
truncations, and size checks pass. The eviction counter is placed
in ReleaseLruFile(), before LruDelete() is called.
---
doc/src/sgml/monitoring.sgml | 148 ++++++++++++++++++
src/backend/catalog/system_views.sql | 18 +++
src/backend/storage/file/fd.c | 49 ++++++
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 17 ++
src/backend/utils/activity/pgstat_vfdcache.c | 154 +++++++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 123 ++++++++++++++-
src/include/catalog/pg_proc.dat | 42 +++++
src/include/pgstat.h | 45 ++++++
src/include/storage/fd.h | 3 +
src/include/utils/pgstat_internal.h | 22 +++
src/include/utils/pgstat_kind.h | 3 +-
src/test/regress/expected/rules.out | 11 ++
src/test/regress/expected/stats.out | 16 +-
src/test/regress/sql/stats.sql | 5 +
16 files changed, 655 insertions(+), 3 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_vfdcache.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bb75ed1069b..0478d7ea50e 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -472,6 +472,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structname>pg_stat_vfdcache</structname>
+ <indexterm><primary>pg_stat_vfdcache</primary></indexterm>
+ </para>
+ <para>
+ One row only, showing cluster-wide statistics about virtual file
+ descriptor (VFD) cache activity. See
+ <link linkend="monitoring-pg-stat-vfdcache-view">
+ <structname>pg_stat_vfdcache</structname></link> for details.
+ </para></entry>
+ </row>
+
<row>
<entry><structname>pg_stat_checkpointer</structname><indexterm><primary>pg_stat_checkpointer</primary></indexterm></entry>
<entry>One row only, showing statistics about the
@@ -3456,6 +3469,128 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</sect2>
+ <sect2 id="monitoring-pg-stat-vfdcache-view">
+ <title><structname>pg_stat_vfdcache</structname></title>
+
+ <indexterm zone="monitoring-pg-stat-vfdcache-view">
+ <primary>pg_stat_vfdcache</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_vfdcache</structname> view will always have a
+ single row, containing data about cluster-wide VFD (Virtual File
+ Descriptor) cache activity.
+ </para>
+
+ <table id="pg-stat-vfdcache-view" xreflabel="pg_stat_vfdcache">
+ <title><structname>pg_stat_vfdcache</structname> View</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry">Column</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hits</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of file accesses where the physical file descriptor was
+ already open in the cache, requiring no system call.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>misses</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of file accesses where the physical file descriptor had
+ been evicted from the cache, requiring <function>open()</function>
+ to be called again before the access could proceed.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>evictions</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of times a physical file descriptor was closed to make
+ room in the cache for a new one (LRU eviction). Each eviction
+ will eventually produce a miss when the evicted file is accessed
+ again. When the cache is thrashing, this value will be close to
+ or equal to <structfield>misses</structfield>.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_entries</structfield> <type>integer</type>
+ </para></entry>
+ <entry><type>integer</type></entry>
+ <entry>
+ Sum of currently allocated VFD cache entries across all active
+ backends.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_bytes</structfield> <type>bigint</type>
+ </para></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Sum of memory used by allocated VFD cache entries across all active
+ backends, in bytes.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>max_files_per_process</structfield> <type>integer</type>
+ </para></entry>
+ <entry><type>integer</type></entry>
+ <entry>
+ Current value of <varname>max_files_per_process</varname>. The
+ effective limit on simultaneously open file descriptors may be
+ lower than this value if the operating system's per-process file
+ descriptor limit is smaller.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hit_ratio</structfield> <type>float8</type>
+ </para></entry>
+ <entry><type>float8</type></entry>
+ <entry>
+ Fraction of file accesses that were cache hits:
+ <literal>hits / (hits + misses)</literal>. A value of
+ <literal>1.0</literal> indicates no cache pressure. Values
+ significantly below <literal>1.0</literal> indicate that
+ <varname>max_files_per_process</varname> should be increased.
+ <literal>NULL</literal> if no accesses have been recorded yet.
+ </entry>
+ </row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield>
+ <type>timestamp with time zone</type>
+ </para></entry>
+ <entry><type>timestamp with time zone</type></entry>
+ <entry>
+ Time at which the counters were last reset by
+ <function>pg_stat_reset_vfdcache()</function>.
+ <literal>NULL</literal> if the counters have never been reset.
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-pg-stat-checkpointer-view">
<title><structname>pg_stat_checkpointer</structname></title>
@@ -5640,6 +5775,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm><primary>pg_stat_reset_vfdcache</primary></indexterm>
+ <function>pg_stat_reset_vfdcache</function> ()
+ <returnvalue>void</returnvalue>
+ </para>
+ <para>
+ Reset shared VFD cache statistics counters to zero. The reset
+ timestamp is recorded in
+ <structname>pg_stat_vfdcache</structname>.<structfield>stats_reset</structfield>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index e54018004db..31d25db6f42 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1516,3 +1516,21 @@ CREATE VIEW pg_aios AS
SELECT * FROM pg_get_aios();
REVOKE ALL ON pg_aios FROM PUBLIC;
GRANT SELECT ON pg_aios TO pg_read_all_stats;
+
+CREATE VIEW pg_stat_vfdcache AS
+ SELECT
+ pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_evictions() AS evictions,
+ pg_stat_get_vfd_cache_size() AS cache_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
+ current_setting('max_files_per_process')::int AS max_files_per_process,
+ CASE
+ WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
+ THEN NULL::float8
+ ELSE pg_stat_get_vfd_hits()::float8
+ / (pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses())
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
+
+GRANT SELECT ON pg_stat_vfdcache TO PUBLIC;
\ No newline at end of file
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 01f1bd6e687..b273c173b11 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1378,6 +1378,7 @@ ReleaseLruFile(void)
* in the ring.
*/
Assert(VfdCache[0].lruMoreRecently != 0);
+ pgstat_count_vfd_eviction();
LruDelete(VfdCache[0].lruMoreRecently);
return true; /* freed a file */
}
@@ -1491,6 +1492,7 @@ FileAccess(File file)
if (FileIsNotOpen(file))
{
+ pgstat_count_vfd_miss();
returnValue = LruInsert(file);
if (returnValue != 0)
return returnValue;
@@ -1501,14 +1503,61 @@ FileAccess(File file)
* We now know that the file is open and that it is not the last one
* accessed, so we need to move it to the head of the Lru ring.
*/
+ pgstat_count_vfd_hit();
Delete(file);
Insert(file);
}
+ else
+ {
+ /* fd is open and already at MRU end */
+ pgstat_count_vfd_hit();
+ }
return 0;
}
+/*
+ * GetVfdCacheOccupancy
+ *
+ * Return the number of physical file descriptors currently open in the
+ * VFD cache (nfile). This is the live cache size exposed by
+ * pg_stat_vfdcache.cache_size.
+ *
+ */
+int
+GetVfdCacheOccupancy(void)
+{
+ return nfile;
+}
+
+/*
+ * GetVfdCacheEntries
+ *
+ * Return the number of VFD cache entries currently allocated for this
+ * backend, excluding slot 0 which is the freelist/LRU header.
+ */
+uint64
+GetVfdCacheEntries(void)
+{
+ if (SizeVfdCache == 0)
+ return 0;
+
+ return (uint64) (SizeVfdCache - 1);
+}
+
+/*
+ * GetVfdCacheBytes
+ *
+ * Return the memory footprint in bytes of currently allocated per-backend
+ * VFD entries, excluding slot 0.
+ */
+uint64
+GetVfdCacheBytes(void)
+{
+ return GetVfdCacheEntries() * sizeof(Vfd);
+}
+
/*
* Called whenever a temporary file is deleted to report its size.
*/
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index ca3ef89bf59..fe8fc00d966 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -32,6 +32,7 @@ OBJS = \
pgstat_shmem.o \
pgstat_slru.o \
pgstat_subscription.o \
+ pgstat_vfdcache.o \
pgstat_wal.o \
pgstat_xact.o \
wait_event.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 1aa7ece5290..5c1bcec7f8b 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -17,6 +17,7 @@ backend_sources += files(
'pgstat_shmem.c',
'pgstat_slru.c',
'pgstat_subscription.c',
+ 'pgstat_vfdcache.c',
'pgstat_wal.c',
'pgstat_xact.c',
)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index eb8ccbaa628..95d366b6ce9 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -500,6 +500,23 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
+
+ [PGSTAT_KIND_VFDCACHE] = {
+ .name = "vfdcache",
+
+ .fixed_amount = true,
+ .write_to_file = true,
+
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, vfdcache),
+ .shared_ctl_off = offsetof(PgStat_ShmemControl, vfdcache),
+ .shared_data_off = offsetof(PgStatShared_VfdCache, stats),
+ .shared_data_len = sizeof(((PgStatShared_VfdCache *) 0)->stats),
+
+ .flush_static_cb = pgstat_vfdcache_flush_cb,
+ .init_shmem_cb = pgstat_vfdcache_init_shmem_cb,
+ .reset_all_cb = pgstat_vfdcache_reset_all_cb,
+ .snapshot_cb = pgstat_vfdcache_snapshot_cb,
+ },
};
/*
diff --git a/src/backend/utils/activity/pgstat_vfdcache.c b/src/backend/utils/activity/pgstat_vfdcache.c
new file mode 100644
index 00000000000..069b833e66e
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_vfdcache.c
@@ -0,0 +1,154 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_vfdcache.c
+ * Implementation of VFD cache statistics.
+ *
+ * VFD events are first counted in backend-local pending storage and then
+ * flushed into shared-memory cumulative stats, following the same model as
+ * other fixed stats kinds.
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_vfdcache.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "pgstat.h"
+#include "storage/fd.h"
+#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+
+/*
+ * Backend-local VFD counters waiting to be flushed.
+ */
+PgStat_VfdCacheStats PendingVfdCacheStats = {0};
+
+/*
+ * Count a VFD cache hit.
+ */
+void
+pgstat_count_vfd_hit(void)
+{
+ PendingVfdCacheStats.vfd_hits++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Count a VFD cache miss.
+ */
+void
+pgstat_count_vfd_miss(void)
+{
+ PendingVfdCacheStats.vfd_misses++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Count a VFD cache eviction.
+ */
+void
+pgstat_count_vfd_eviction(void)
+{
+ PendingVfdCacheStats.vfd_evictions++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Flush out backend-local pending VFD cache stats.
+ */
+bool
+pgstat_vfdcache_flush_cb(bool nowait)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_Backend *shbackendent;
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ if (pg_memory_is_all_zeros(&PendingVfdCacheStats,
+ sizeof(struct PgStat_VfdCacheStats)))
+ return false;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
+ MyProcNumber, nowait);
+ if (!entry_ref)
+ return true;
+
+ shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
+
+ shbackendent->stats.vfd_stats.vfd_entries = (PgStat_Counter) GetVfdCacheEntries();
+ shbackendent->stats.vfd_stats.vfd_cache_bytes = (PgStat_Counter) GetVfdCacheBytes();
+
+ pgstat_unlock_entry(entry_ref);
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.vfd_hits += PendingVfdCacheStats.vfd_hits;
+ stats_shmem->stats.vfd_misses += PendingVfdCacheStats.vfd_misses;
+ stats_shmem->stats.vfd_evictions += PendingVfdCacheStats.vfd_evictions;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ MemSet(&PendingVfdCacheStats, 0, sizeof(PendingVfdCacheStats));
+
+ return false;
+}
+
+/*
+ * Support function for SQL-callable pg_stat_get_vfd_* functions.
+ */
+PgStat_VfdCacheStats *
+pgstat_fetch_stat_vfdcache(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_VFDCACHE);
+
+ return &pgStatLocal.snapshot.vfdcache;
+}
+
+void
+pgstat_vfdcache_init_shmem_cb(void *stats)
+{
+ PgStatShared_VfdCache *stats_shmem = (PgStatShared_VfdCache *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+void
+pgstat_vfdcache_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_vfdcache_snapshot_cb(void)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+ PgStat_VfdCacheStats *reset_offset = &stats_shmem->reset_offset;
+ PgStat_VfdCacheStats reset;
+
+ pgstat_copy_changecounted_stats(&pgStatLocal.snapshot.vfdcache,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ pgStatLocal.snapshot.vfdcache.vfd_hits -= reset.vfd_hits;
+ pgStatLocal.snapshot.vfdcache.vfd_misses -= reset.vfd_misses;
+ pgStatLocal.snapshot.vfdcache.vfd_evictions -= reset.vfd_evictions;
+}
+
+void
+pgstat_reset_vfdcache(void)
+{
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 9185a8e6b83..0f36acac56c 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "access/xact.h"
#include "access/htup_details.h"
#include "access/xlog.h"
#include "access/xlogprefetcher.h"
@@ -28,6 +29,7 @@
#include "replication/logicallauncher.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/fd.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/timestamp.h"
@@ -1340,6 +1342,122 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+/*
+ * Sum per-backend VFD gauges across currently active backends.
+ */
+static void
+pgstat_get_vfd_backend_sums(PgStat_Counter *entries_sum,
+ PgStat_Counter *bytes_sum)
+{
+ static TimestampTz cached_stmt_start_ts = 0;
+ static PgStat_Counter cached_entries_sum = 0;
+ static PgStat_Counter cached_bytes_sum = 0;
+ TimestampTz stmt_start_ts = GetCurrentStatementStartTimestamp();
+ int num_backends = pgstat_fetch_stat_numbackends();
+
+ if (cached_stmt_start_ts == stmt_start_ts)
+ {
+ *entries_sum = cached_entries_sum;
+ *bytes_sum = cached_bytes_sum;
+ return;
+ }
+
+ *entries_sum = 0;
+ *bytes_sum = 0;
+
+ for (int curr_backend = 1; curr_backend <= num_backends; curr_backend++)
+ {
+ LocalPgBackendStatus *local_beentry;
+ PgBackendStatus *beentry;
+ PgStat_Backend *backend_stats;
+
+ local_beentry = pgstat_get_local_beentry_by_index(curr_backend);
+ beentry = &local_beentry->backendStatus;
+
+ if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
+ continue;
+
+ backend_stats = pgstat_fetch_stat_backend(local_beentry->proc_number);
+ if (!backend_stats)
+ continue;
+
+ *entries_sum += backend_stats->vfd_stats.vfd_entries;
+ *bytes_sum += backend_stats->vfd_stats.vfd_cache_bytes;
+ }
+
+ cached_entries_sum = *entries_sum;
+ cached_bytes_sum = *bytes_sum;
+ cached_stmt_start_ts = stmt_start_ts;
+}
+
+Datum
+pg_stat_get_vfd_hits(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_hits);
+}
+
+Datum
+pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
+}
+
+Datum
+pg_stat_get_vfd_evictions(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_evictions);
+}
+
+Datum
+pg_stat_get_vfd_cache_size(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter entries_sum;
+ PgStat_Counter bytes_dummy;
+
+ pgstat_get_vfd_backend_sums(&entries_sum, &bytes_dummy);
+
+ PG_RETURN_INT32((int32) entries_sum);
+}
+
+Datum
+pg_stat_get_vfd_cache_bytes(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter entries_sum;
+ PgStat_Counter bytes_sum;
+
+ pgstat_get_vfd_backend_sums(&entries_sum, &bytes_sum);
+
+ PG_RETURN_INT64(bytes_sum);
+}
+
+/*
+ * pg_stat_get_vfd_stat_reset_time
+ * Timestamp of the last pg_stat_reset_vfdcache() call, or NULL if
+ * the counters have never been reset.
+ */
+Datum
+pg_stat_get_vfd_stat_reset_time(PG_FUNCTION_ARGS)
+{
+ TimestampTz ts = pgstat_fetch_stat_vfdcache()->stat_reset_timestamp;
+
+ if (ts == 0)
+ PG_RETURN_NULL();
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+}
+
+/*
+ * pg_stat_reset_vfdcache
+ * Reset shared VFD cache counters.
+ */
+Datum
+pg_stat_reset_vfdcache(PG_FUNCTION_ARGS)
+{
+ pgstat_reset_vfdcache();
+ PG_RETURN_VOID();
+}
+
+
/*
* When adding a new column to the pg_stat_io view and the
* pg_stat_get_backend_io() function, add a new enum value here above
@@ -1965,6 +2083,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
PG_RETURN_VOID();
}
@@ -1987,11 +2106,13 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
else if (strcmp(target, "wal") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
+ else if (strcmp(target, "vfdcache") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", \"vfdcache\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0118e970dda..b6930174afb 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12850,5 +12850,47 @@
{ oid => '8281', descr => 'hash',
proname => 'hashoid8extended', prorettype => 'int8',
proargtypes => 'oid8 int8', prosrc => 'hashoid8extended' },
+{ oid => '9560',
+ descr => 'statistics: number of VFD cache hits',
+ proname => 'pg_stat_get_vfd_hits',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_hits' },
+{ oid => '9561',
+ descr => 'statistics: number of VFD cache misses',
+ proname => 'pg_stat_get_vfd_misses',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_misses' },
+{ oid => '9562',
+ descr => 'statistics: number of VFD cache evictions',
+ proname => 'pg_stat_get_vfd_evictions',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_evictions' },
+{ oid => '9563',
+ descr => 'statistics: total number of allocated VFD cache entries',
+ proname => 'pg_stat_get_vfd_cache_size',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int4', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_size' },
+{ oid => '9566',
+ descr => 'statistics: total memory footprint of VFD cache entries in bytes',
+ proname => 'pg_stat_get_vfd_cache_bytes',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_bytes' },
+{ oid => '9564',
+ descr => 'statistics: timestamp of last VFD cache stats reset',
+ proname => 'pg_stat_get_vfd_stat_reset_time',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'timestamptz', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_stat_reset_time' },
+{ oid => '9565',
+ descr => 'statistics: reset shared VFD cache counters',
+ proname => 'pg_stat_reset_vfdcache',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'void', proargtypes => '',
+ prosrc => 'pg_stat_reset_vfdcache' },
]
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 8e3549c3752..7e7e5b9383e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -272,6 +272,35 @@ typedef struct PgStat_CheckpointerStats
TimestampTz stat_reset_timestamp;
} PgStat_CheckpointerStats;
+/* ---------
+ * PgStat_VfdCacheStats Virtual File Descriptor cache statistics
+ *
+ * Tracks hit/miss/eviction events in the VFD cache (fd.c). These counters
+ * are accumulated in shared fixed stats and exposed by pg_stat_vfdcache.
+ * ---------
+ */
+typedef struct PgStat_VfdCacheStats
+{
+ PgStat_Counter vfd_hits; /* fd was open, no open() syscall needed */
+ PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was required */
+ PgStat_Counter vfd_evictions; /* close() called to free a slot for a new
+ * fd */
+ TimestampTz stat_reset_timestamp;
+} PgStat_VfdCacheStats;
+
+/* ---------
+ * PgStat_BackendVfdCacheStats VFD cache stats stored per backend
+ *
+ * Keeps per-backend VFD gauges in PGSTAT_KIND_BACKEND entries.
+ * vfd_entries and vfd_cache_bytes represent the current backend-local VFD
+ * cache footprint and are used to derive cluster-wide totals.
+ * ---------
+ */
+typedef struct PgStat_BackendVfdCacheStats
+{
+ PgStat_Counter vfd_entries;
+ PgStat_Counter vfd_cache_bytes;
+} PgStat_BackendVfdCacheStats;
/*
* Types related to counting IO operations
@@ -521,6 +550,7 @@ typedef struct PgStat_WalStats
typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
+ PgStat_BackendVfdCacheStats vfd_stats;
PgStat_BktypeIO io_stats;
PgStat_WalCounters wal_counters;
} PgStat_Backend;
@@ -611,6 +641,15 @@ extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void);
extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern PgStat_VfdCacheStats * pgstat_fetch_stat_vfdcache(void);
+extern void pgstat_reset_vfdcache(void);
+extern void pgstat_count_vfd_hit(void);
+extern void pgstat_count_vfd_miss(void);
+extern void pgstat_count_vfd_eviction(void);
/*
* Functions in pgstat_io.c
@@ -851,6 +890,12 @@ extern PGDLLIMPORT int pgstat_fetch_consistency;
/* updated directly by bgwriter and bufmgr */
extern PGDLLIMPORT PgStat_BgWriterStats PendingBgWriterStats;
+/*
+ * Variables in pgstat_vfdcache.c
+ */
+
+/* updated by VFD counting functions called from fd.c */
+extern PGDLLIMPORT PgStat_VfdCacheStats PendingVfdCacheStats;
/*
* Variables in pgstat_checkpointer.c
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 8ac466fd346..e666a59f18d 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -149,6 +149,9 @@ extern char *FilePathName(File file);
extern int FileGetRawDesc(File file);
extern int FileGetRawFlags(File file);
extern mode_t FileGetRawMode(File file);
+extern int GetVfdCacheOccupancy(void);
+extern uint64 GetVfdCacheEntries(void);
+extern uint64 GetVfdCacheBytes(void);
/* Operations used for sharing named temporary files */
extern File PathNameCreateTemporaryFile(const char *path, bool error_on_failure);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 97704421a92..98d3f8ffbf9 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -488,6 +488,15 @@ typedef struct PgStatShared_Wal
PgStat_WalStats stats;
} PgStatShared_Wal;
+typedef struct PgStatShared_VfdCache
+{
+ /* lock protects ->reset_offset as well as stats->stat_reset_timestamp */
+ LWLock lock;
+ uint32 changecount;
+ PgStat_VfdCacheStats stats;
+ PgStat_VfdCacheStats reset_offset;
+} PgStatShared_VfdCache;
+
/* ----------
@@ -583,6 +592,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Lock lock;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+ PgStatShared_VfdCache vfdcache;
/*
* Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
@@ -619,6 +629,8 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ PgStat_VfdCacheStats vfdcache;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -732,6 +744,16 @@ extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern bool pgstat_vfdcache_flush_cb(bool nowait);
+extern void pgstat_vfdcache_init_shmem_cb(void *stats);
+extern void pgstat_vfdcache_reset_all_cb(TimestampTz ts);
+extern void pgstat_vfdcache_snapshot_cb(void);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 2d78a029683..319ec0bd63b 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -39,9 +39,10 @@
#define PGSTAT_KIND_LOCK 11
#define PGSTAT_KIND_SLRU 12
#define PGSTAT_KIND_WAL 13
+#define PGSTAT_KIND_VFDCACHE 14
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
+#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_VFDCACHE
#define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1)
/* Custom stats kinds */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2b3cf6d8569..40320652624 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2362,6 +2362,17 @@ pg_stat_user_tables| SELECT relid,
stats_reset
FROM pg_stat_all_tables
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
+pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_evictions() AS evictions,
+ pg_stat_get_vfd_cache_size() AS cache_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
+ (current_setting('max_files_per_process'::text))::integer AS max_files_per_process,
+ CASE
+ WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
+ ELSE ((pg_stat_get_vfd_hits())::double precision / ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()))::double precision)
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index ea7f7846895..2c168a40ed9 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1127,10 +1127,24 @@ SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
t
(1 row)
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+ ?column?
+----------
+ t
+(1 row)
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
ERROR: unrecognized reset target: "unknown"
-HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", or "wal".
+HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", "vfdcache", or "wal".
-- Test that reset works for pg_stat_database and pg_stat_database_conflicts
-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so that we
-- have a baseline for comparison. The same for pg_stat_database_conflicts as it shares
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 65d8968c83e..0b486e0f884 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -520,6 +520,11 @@ SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset
SELECT pg_stat_reset_shared('wal');
SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
--
2.34.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-03-31 14:54 ` David Geier <geidav.pg@gmail.com>
1 sibling, 0 replies; 28+ messages in thread
From: David Geier @ 2026-03-31 14:54 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers; tomas@vondra.me <tomas@vondra.me>
Hi!
On 29.03.2026 21:46, KAZAR Ayoub wrote:
>>>>> I've looked at struct vfd and some simple changes to the struct would
>>>>> already cut memory consumption in half. I can look into that.
>>>>>
>>>>> Thoughts?
>>>>
>>>> Looking forward to this.
>>>
>>> I try to come up with something the next days.
>>>> What also bothers me in that space is if a backend allocates 100K
>>> entries
>>>> in VFD cache, that cache is never shrank ever again,
>>>> the cache only grows (if it needs more than its lifetime maximum) until
>>> the
>>>> backend dies, although this is useful as entries are reused if free
>>> instead
>>>> of
>>>> allocating entries, whether a spike in files openings effects a long
>>> living
>>>> backend to keep holding a useless amount of
>>>> cache size it will need in the future, i don't imagine this to be common
>>>> though, what do you think about this issue from your experience ?
>>>
>>> Currently the cache is directly mapped by the VFD index. That means we
>>> could only resize down to the maximum used VFD index.
>>>
>>> Being able to resize independently of the maximum VFD index would
>>> require changing to a hash map like simplehash.h. I can take a look how
>>> invasive such a change would be.
That would actually be doable without too much code churn. It would,
however, add some more overhead to each cache entry: 1 byte for the
simplehash.h status and a 4 byte for the hash, if we want to avoid
rehashing on each access. Probably we can get away without storing the
hash. We would then have to monitor the cache size and recreate the hash
table when the size has shrunk by enough.
Another alternative is pallocing vfd entries so that we can freely move
them around in the vfd cache array. That would mean an ABI change (File
would be an 8-byte pointer instead of a 4-byte integer) but give us much
more flexibility for possible improvements.
For example, apart from compacting without hash map, this would allow us
to allocate variable amounts of memory per entry to, e.g.
- store the file name inline of the struct as variable length array
(instead of a pstrdup() pointer) and
- depending on type allocate more or less memory. This is because
non-temporary files don't use the ResOwner and the file size.
>> I've implemented the recommended global stats view on vfd cache, the
>> implementation should be also straightforward as it follows the same
>> cumulative shared statistics infrastructure like pgstat_bgwriter and others
>> do.
>>
>> Attached is v2 patch also contains what David suggested for global cache
>> size and entries in the view.
I'll review the patch the next days. On quick inspection I saw that you
compute the size of a struct vfd by using sizeof(). That works, except
for the filename which is stored as a pointer to a pstrdup() piece of
memory. I guess you can just say:
sizeof(VfdCache[i]) + GetMemoryChunkSpace(VfdCache[i].fileName)
--
David Geier
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-03-31 18:27 ` Tomas Vondra <tomas@vondra.me>
2026-04-03 13:53 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 1 reply; 28+ messages in thread
From: Tomas Vondra @ 2026-03-31 18:27 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; David Geier <geidav.pg@gmail.com>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers
Hi,
Thanks for working on this, I think having some stats about the vfd
cache would be quite helpful. I took a quick look at the patch, and in
general it goes in the right direction.
Here's a couple comments / suggestions:
1) sgml docs
- Formatting seems a bit wrong, both for the entry in the first table,
and then for the view description later. Clearly different from the
nearby tables after the documentation is built. Did you generate the
tables somehow?
- The ordering seems a bit random, but I'd argue the pg_stat_vfdcache
view should go before pg_stat_wal (at least in the first table).
2) system_views.sql
- missing the REVOKE command
- We don't align the AS clauses with spaces (I like to align my queries,
but here it's a question of consistency with the other commands, and the
alignment makes future diffs larger)
3) fd.c
- Wouldn't it be better to have pgstat_count_vfd_access(hit bool)? That
way you would't even need the new else, it'd ne enough to do
pgstat_count_vfd_access(!FileIsNotOpen(file)) at the beginning.
- GetVfdCacheOccupancy does not seem to be called from anywhere
4) pgstat_vfdcache.c
- I'm not sure usagecount makes sense for these stats, because that's
for cases with a single writer. These stats are written by backends, so
it probably needs a lwlock.
- I'm not sure updating PgStatShared_Backend from pgstat_vfdcache.c is a
good idea, when it's already synced from pgstat_backend. We don't do
that for WAL either, but there's pgstat_flush_backend_entry_wal in
pgstat_backend.c. I suppose vfdcache should do it the same way.
5) pg_proc.dat
- formatting seems a bit inconsistent
6) pgstat.h
- Aren't evictions mostly the same as misses, at least after a while?
- I think it would be useful to report how many file descriptors we
are allowed to open (it's less than max_files_per_process, depending
on the ulimits etc.)
- I know io_uring can consume quite a few descriptors, and it can cause
issues, I wonder if this would make it easier to observe
I also suggest to split the patch into smaller patches, to make it
easier to review and evaluate. Not because of size - the patch is fairly
small. But it's better to not mix multiple features with different
cost/benefit trade offs, because then it's possible to evaluate them
separately. Maybe even commit the first part and continue discussion
about the following one(s).
This patch seems to mix two different types of stats - global stats of
the vfd cache, and then also per-backend stats. Those seems like very
different things, both in terms of overhead and benefits.
The global cache stats is going to be virtually free (at least the
hits/misses, I'm not sure about the number of entries and bytes), and
it's obviously useful for tuning the max_files_per_process GUC. I'd even
contemplate getting this into PG19, maybe.
The per-backend stats seem like a much harder sell to me, but I can be
convinced. Maybe it's not an issue in terms of overhead, maybe the stats
we get from that are worth it. Not sure. But I'd keep it in a separate
0002 patch, on top of 0001 with just the "global" stats.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-31 18:27 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Tomas Vondra <tomas@vondra.me>
@ 2026-04-03 13:53 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-04-23 07:34 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
0 siblings, 1 reply; 28+ messages in thread
From: KAZAR Ayoub @ 2026-04-03 13:53 UTC (permalink / raw)
To: Tomas Vondra <tomas@vondra.me>; +Cc: David Geier <geidav.pg@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers
Hello,
Thanks for the review!
On Tue, Mar 31, 2026 at 8:27 PM Tomas Vondra <tomas@vondra.me> wrote:
> Hi,
>
> Thanks for working on this, I think having some stats about the vfd
> cache would be quite helpful. I took a quick look at the patch, and in
> general it goes in the right direction.
>
> Here's a couple comments / suggestions:
>
>
6) pgstat.h
>
> - Aren't evictions mostly the same as misses, at least after a while?
>
Correct, the time where they are not the same is pretty much meaningless
info, i removed it.
>
> - I think it would be useful to report how many file descriptors we
> are allowed to open (it's less than max_files_per_process, depending
> on the ulimits etc.)
>
Agree, This should be max_safe_fds calculated by postmaster, I added this
but let me know if its acceptable to export max_safe_fds in the way I did.
>
> - I know io_uring can consume quite a few descriptors, and it can cause
> issues, I wonder if this would make it easier to observe
>
>
> I also suggest to split the patch into smaller patches, to make it
> easier to review and evaluate. Not because of size - the patch is fairly
> small. But it's better to not mix multiple features with different
> cost/benefit trade offs, because then it's possible to evaluate them
> separately. Maybe even commit the first part and continue discussion
> about the following one(s).
>
> This patch seems to mix two different types of stats - global stats of
> the vfd cache, and then also per-backend stats. Those seems like very
> different things, both in terms of overhead and benefits.
>
> The global cache stats is going to be virtually free (at least the
> hits/misses, I'm not sure about the number of entries and bytes), and
> it's obviously useful for tuning the max_files_per_process GUC. I'd even
> contemplate getting this into PG19, maybe.
>
> The per-backend stats seem like a much harder sell to me, but I can be
> convinced. Maybe it's not an issue in terms of overhead, maybe the stats
> we get from that are worth it. Not sure. But I'd keep it in a separate
> 0002 patch, on top of 0001 with just the "global" stats.
>
>
>
> regards
>
> --
> Tomas Vondra
>
> I fixed style related issues and followed your suggestions on splitting
the patch to do global stats first then the per-backend stats of cache size
and entries count reporting, attached is v3-0001 that does just the global
stats counting.
When we make sure this is correct i'll proceed with the per-backend stats
patch.
Regards,
Ayoub
Attachments:
[text/x-patch] v3-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch (24.1K, ../../CA+K2RumSp-kTw_YHXs_qN_RLt6cWfFR=LMq9coLgu8eyGydpHQ@mail.gmail.com/3-v3-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch)
download | inline diff:
From 91cf8d91a2e57ae6dde5291660439ca4ce0a6bc5 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Fri, 3 Apr 2026 12:49:42 +0200
Subject: [PATCH] Add pg_stat_vfdcache view for VFD cache statistics
PostgreSQL's virtual file descriptor (VFD) layer maintains a
per-backend cache of open file descriptors bounded by
max_files_per_process (default 1000). When the cache is full, the
least-recently-used entry is evicted (its OS fd closed) so a new file
can be opened. A subsequent access to an evicted file must call
open() again.
A trivial example is with partitioned tables: a table with 1500
partitions requires up to many file descriptors per full scan (main
fork, vm ...), which is more than the default limit, causing
potential evictions and reopens.
This commit adds:
pg_stat_vfdcache -- a single-row view exposing cluster-wide VFD cache
statistics:
hits number of VFD cache hits
misses number of VFD cache misses
max_open_fds maximum number of file descriptors available to each backend process
hit_ratio hits / (hits + misses)
stats_reset timestamp of last counter reset
pg_stat_reset_vfdcache() -- resets shared VFD counters
The implementation follows the same cumulative shared statistics infrastructure like pgstat_bgwriter and others do.
Event counting remains cheap in backend-local pending storage and is flushed
into shared fixed stats which requires locking.
Hit and miss counters are placed in FileAccess(), which is the
single gate through which all VFD-mediated file reads, writes,
truncations, and size checks pass.
---
doc/src/sgml/monitoring.sgml | 107 ++++++++++++++++++
src/backend/catalog/system_views.sql | 16 +++
src/backend/storage/file/fd.c | 5 +-
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 17 +++
src/backend/utils/activity/pgstat_vfdcache.c | 113 +++++++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 47 +++++++-
src/include/catalog/pg_proc.dat | 30 +++++
src/include/pgstat.h | 26 +++++
src/include/utils/pgstat_internal.h | 20 ++++
src/include/utils/pgstat_kind.h | 3 +-
src/test/regress/expected/rules.out | 8 ++
src/test/regress/expected/stats.out | 16 ++-
src/test/regress/sql/stats.sql | 5 +
15 files changed, 411 insertions(+), 4 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_vfdcache.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bb75ed1069b..cab7d4457e6 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -543,6 +543,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_vfdcache</structname><indexterm><primary>pg_stat_vfdcache</primary></indexterm></entry>
+ <entry>One row only, showing cluster-wide statistics about virtual file
+ descriptor (VFD) cache activity. See
+ <link linkend="monitoring-pg-stat-vfdcache-view">
+ <structname>pg_stat_vfdcache</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_wal</structname><indexterm><primary>pg_stat_wal</primary></indexterm></entry>
<entry>One row only, showing statistics about WAL activity. See
@@ -3603,6 +3612,89 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
</sect2>
+ <sect2 id="monitoring-pg-stat-vfdcache-view">
+ <title><structname>pg_stat_vfdcache</structname></title>
+
+ <indexterm zone="monitoring-pg-stat-vfdcache-view">
+ <primary>pg_stat_vfdcache</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_vfdcache</structname> view will always have a
+ single row, containing data about cluster-wide VFD (Virtual File
+ Descriptor) cache activity.
+ </para>
+
+ <table id="pg-stat-vfdcache-view" xreflabel="pg_stat_vfdcache">
+ <title><structname>pg_stat_vfdcache</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hits</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of file accesses where the physical file descriptor was
+ already open in the cache, requiring no system call
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>misses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of file accesses where the physical file descriptor had
+ been evicted from the cache, requiring <function>open()</function>
+ to be called again before the access could proceed
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>max_open_fds</structfield> <type>integer</type>
+ </para>
+ <para>
+ Maximum number of file descriptors available to each backend process
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hit_ratio</structfield> <type>float8</type>
+ </para>
+ <para>
+ Fraction of file accesses that were cache hits:
+ <literal>hits / (hits + misses)</literal>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which the counters were last reset by
+ <function>pg_stat_reset_vfdcache()</function>
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-pg-stat-wal-view">
<title><structname>pg_stat_wal</structname></title>
@@ -5640,6 +5732,21 @@ description | Waiting for a newly initialized WAL file to reach durable storage
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_stat_reset_vfdcache</primary>
+ </indexterm>
+ <function>pg_stat_reset_vfdcache</function> ()
+ <returnvalue>void</returnvalue>
+ </para>
+ <para>
+ Reset shared VFD cache statistics counters to zero. The reset
+ timestamp is recorded in
+ <structname>pg_stat_vfdcache</structname>.<structfield>stats_reset</structfield>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index e54018004db..34839e60146 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1516,3 +1516,19 @@ CREATE VIEW pg_aios AS
SELECT * FROM pg_get_aios();
REVOKE ALL ON pg_aios FROM PUBLIC;
GRANT SELECT ON pg_aios TO pg_read_all_stats;
+
+CREATE VIEW pg_stat_vfdcache AS
+ SELECT
+ pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_max_open_fds() AS max_open_fds,
+ CASE
+ WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
+ THEN NULL::float8
+ ELSE pg_stat_get_vfd_hits()::float8
+ / (pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses())
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
+
+REVOKE ALL ON pg_stat_vfdcache FROM PUBLIC;
+GRANT SELECT ON pg_stat_vfdcache TO PUBLIC;
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 01f1bd6e687..8427e98ff75 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1480,6 +1480,7 @@ static int
FileAccess(File file)
{
int returnValue;
+ bool is_open;
DO_DB(elog(LOG, "FileAccess %d (%s)",
file, VfdCache[file].fileName));
@@ -1488,8 +1489,10 @@ FileAccess(File file)
* Is the file open? If not, open it and put it at the head of the LRU
* ring (possibly closing the least recently used file to get an FD).
*/
+ is_open = !FileIsNotOpen(file);
+ pgstat_count_vfd_access(is_open);
- if (FileIsNotOpen(file))
+ if (!is_open)
{
returnValue = LruInsert(file);
if (returnValue != 0)
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index ca3ef89bf59..fe8fc00d966 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -32,6 +32,7 @@ OBJS = \
pgstat_shmem.o \
pgstat_slru.o \
pgstat_subscription.o \
+ pgstat_vfdcache.o \
pgstat_wal.o \
pgstat_xact.o \
wait_event.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 1aa7ece5290..5c1bcec7f8b 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -17,6 +17,7 @@ backend_sources += files(
'pgstat_shmem.c',
'pgstat_slru.c',
'pgstat_subscription.c',
+ 'pgstat_vfdcache.c',
'pgstat_wal.c',
'pgstat_xact.c',
)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index eb8ccbaa628..95d366b6ce9 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -500,6 +500,23 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
+
+ [PGSTAT_KIND_VFDCACHE] = {
+ .name = "vfdcache",
+
+ .fixed_amount = true,
+ .write_to_file = true,
+
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, vfdcache),
+ .shared_ctl_off = offsetof(PgStat_ShmemControl, vfdcache),
+ .shared_data_off = offsetof(PgStatShared_VfdCache, stats),
+ .shared_data_len = sizeof(((PgStatShared_VfdCache *) 0)->stats),
+
+ .flush_static_cb = pgstat_vfdcache_flush_cb,
+ .init_shmem_cb = pgstat_vfdcache_init_shmem_cb,
+ .reset_all_cb = pgstat_vfdcache_reset_all_cb,
+ .snapshot_cb = pgstat_vfdcache_snapshot_cb,
+ },
};
/*
diff --git a/src/backend/utils/activity/pgstat_vfdcache.c b/src/backend/utils/activity/pgstat_vfdcache.c
new file mode 100644
index 00000000000..b036759031c
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_vfdcache.c
@@ -0,0 +1,113 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_vfdcache.c
+ * Implementation of VFD cache statistics.
+ *
+ * VFD events are first counted in backend-local pending storage and then
+ * flushed into shared-memory cumulative stats, following the same model as
+ * other fixed stats kinds.
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_vfdcache.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "pgstat.h"
+#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+
+/*
+ * Backend-local VFD counters waiting to be flushed.
+ */
+PgStat_VfdCacheStats PendingVfdCacheStats = {0};
+
+/*
+ * Count a VFD cache access as either a hit or miss.
+ */
+void
+pgstat_count_vfd_access(bool hit)
+{
+ if (hit)
+ PendingVfdCacheStats.vfd_hits++;
+ else
+ PendingVfdCacheStats.vfd_misses++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Flush out backend-local pending VFD cache stats.
+ */
+bool
+pgstat_vfdcache_flush_cb(bool nowait)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ if (pg_memory_is_all_zeros(&PendingVfdCacheStats,
+ sizeof(struct PgStat_VfdCacheStats)))
+ return false;
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ stats_shmem->stats.vfd_hits += PendingVfdCacheStats.vfd_hits;
+ stats_shmem->stats.vfd_misses += PendingVfdCacheStats.vfd_misses;
+
+ LWLockRelease(&stats_shmem->lock);
+
+ MemSet(&PendingVfdCacheStats, 0, sizeof(PendingVfdCacheStats));
+
+ return false;
+}
+
+/*
+ * Support function for SQL-callable pg_stat_get_vfd_* functions.
+ */
+PgStat_VfdCacheStats *
+pgstat_fetch_stat_vfdcache(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_VFDCACHE);
+
+ return &pgStatLocal.snapshot.vfdcache;
+}
+
+void
+pgstat_vfdcache_init_shmem_cb(void *stats)
+{
+ PgStatShared_VfdCache *stats_shmem = (PgStatShared_VfdCache *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+void
+pgstat_vfdcache_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ MemSet(&stats_shmem->stats, 0, sizeof(stats_shmem->stats));
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_vfdcache_snapshot_cb(void)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&pgStatLocal.snapshot.vfdcache, &stats_shmem->stats,
+ sizeof(pgStatLocal.snapshot.vfdcache));
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_reset_vfdcache(void)
+{
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 9185a8e6b83..3c9e78049ba 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -26,6 +26,7 @@
#include "pgstat.h"
#include "postmaster/bgworker.h"
#include "replication/logicallauncher.h"
+#include "storage/fd.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/acl.h"
@@ -1340,6 +1341,47 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+Datum
+pg_stat_get_vfd_hits(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_hits);
+}
+
+Datum
+pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
+}
+
+Datum
+pg_stat_get_vfd_max_open_fds(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT32(max_safe_fds);
+}
+
+Datum
+pg_stat_get_vfd_stat_reset_time(PG_FUNCTION_ARGS)
+{
+ TimestampTz ts = pgstat_fetch_stat_vfdcache()->stat_reset_timestamp;
+
+ if (ts == 0)
+ PG_RETURN_NULL();
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+}
+
+/*
+ * pg_stat_reset_vfdcache
+ * Reset shared VFD cache counters.
+ */
+Datum
+pg_stat_reset_vfdcache(PG_FUNCTION_ARGS)
+{
+ pgstat_reset_vfdcache();
+ PG_RETURN_VOID();
+}
+
+
/*
* When adding a new column to the pg_stat_io view and the
* pg_stat_get_backend_io() function, add a new enum value here above
@@ -1965,6 +2007,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
PG_RETURN_VOID();
}
@@ -1985,13 +2028,15 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
else if (strcmp(target, "slru") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
+ else if (strcmp(target, "vfdcache") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
else if (strcmp(target, "wal") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", \"vfdcache\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index acf16254b21..16adfd5c731 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12859,5 +12859,35 @@
{ oid => '8281', descr => 'hash',
proname => 'hashoid8extended', prorettype => 'int8',
proargtypes => 'oid8 int8', prosrc => 'hashoid8extended' },
+{ oid => '9560',
+ descr => 'statistics: number of VFD cache hits',
+ proname => 'pg_stat_get_vfd_hits',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_hits' },
+{ oid => '9561',
+ descr => 'statistics: number of VFD cache misses',
+ proname => 'pg_stat_get_vfd_misses',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_misses' },
+{ oid => '9564',
+ descr => 'statistics: timestamp of last VFD cache stats reset',
+ proname => 'pg_stat_get_vfd_stat_reset_time',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'timestamptz', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_stat_reset_time' },
+{ oid => '9565',
+ descr => 'statistics: reset shared VFD cache counters',
+ proname => 'pg_stat_reset_vfdcache',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'void', proargtypes => '',
+ prosrc => 'pg_stat_reset_vfdcache' },
+{ oid => '9566',
+ descr => 'statistics: max number of file descriptors available to backend',
+ proname => 'pg_stat_get_vfd_max_open_fds',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int4', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_max_open_fds' },
]
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 8e3549c3752..ccb3ba28286 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -272,6 +272,19 @@ typedef struct PgStat_CheckpointerStats
TimestampTz stat_reset_timestamp;
} PgStat_CheckpointerStats;
+/* ---------
+ * PgStat_VfdCacheStats Virtual File Descriptor cache statistics
+ *
+ * Tracks hit/miss events in the VFD cache (fd.c). These counters
+ * are accumulated in shared fixed stats and exposed by pg_stat_vfdcache.
+ * ---------
+ */
+typedef struct PgStat_VfdCacheStats
+{
+ PgStat_Counter vfd_hits; /* fd was open, no open() was needed */
+ PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was required */
+ TimestampTz stat_reset_timestamp;
+} PgStat_VfdCacheStats;
/*
* Types related to counting IO operations
@@ -611,6 +624,13 @@ extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void);
extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern PgStat_VfdCacheStats * pgstat_fetch_stat_vfdcache(void);
+extern void pgstat_reset_vfdcache(void);
+extern void pgstat_count_vfd_access(bool hit);
/*
* Functions in pgstat_io.c
@@ -851,6 +871,12 @@ extern PGDLLIMPORT int pgstat_fetch_consistency;
/* updated directly by bgwriter and bufmgr */
extern PGDLLIMPORT PgStat_BgWriterStats PendingBgWriterStats;
+/*
+ * Variables in pgstat_vfdcache.c
+ */
+
+/* updated by VFD counting functions called from fd.c */
+extern PGDLLIMPORT PgStat_VfdCacheStats PendingVfdCacheStats;
/*
* Variables in pgstat_checkpointer.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index eed4c6b359c..41cd562fa74 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -488,6 +488,13 @@ typedef struct PgStatShared_Wal
PgStat_WalStats stats;
} PgStatShared_Wal;
+typedef struct PgStatShared_VfdCache
+{
+ /* lock protects ->stats */
+ LWLock lock;
+ PgStat_VfdCacheStats stats;
+} PgStatShared_VfdCache;
+
/* ----------
@@ -583,6 +590,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Lock lock;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+ PgStatShared_VfdCache vfdcache;
/*
* Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
@@ -619,6 +627,8 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ PgStat_VfdCacheStats vfdcache;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -732,6 +742,16 @@ extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern bool pgstat_vfdcache_flush_cb(bool nowait);
+extern void pgstat_vfdcache_init_shmem_cb(void *stats);
+extern void pgstat_vfdcache_reset_all_cb(TimestampTz ts);
+extern void pgstat_vfdcache_snapshot_cb(void);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 2d78a029683..319ec0bd63b 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -39,9 +39,10 @@
#define PGSTAT_KIND_LOCK 11
#define PGSTAT_KIND_SLRU 12
#define PGSTAT_KIND_WAL 13
+#define PGSTAT_KIND_VFDCACHE 14
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
+#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_VFDCACHE
#define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1)
/* Custom stats kinds */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2b3cf6d8569..9c0e4708285 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2362,6 +2362,14 @@ pg_stat_user_tables| SELECT relid,
stats_reset
FROM pg_stat_all_tables
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
+pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_max_open_fds() AS max_open_fds,
+ CASE
+ WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
+ ELSE ((pg_stat_get_vfd_hits())::double precision / ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()))::double precision)
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index ea7f7846895..2c168a40ed9 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1127,10 +1127,24 @@ SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
t
(1 row)
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+ ?column?
+----------
+ t
+(1 row)
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
ERROR: unrecognized reset target: "unknown"
-HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", or "wal".
+HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", "vfdcache", or "wal".
-- Test that reset works for pg_stat_database and pg_stat_database_conflicts
-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so that we
-- have a baseline for comparison. The same for pg_stat_database_conflicts as it shares
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 65d8968c83e..0b486e0f884 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -520,6 +520,11 @@ SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset
SELECT pg_stat_reset_shared('wal');
SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
--
2.34.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-31 18:27 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Tomas Vondra <tomas@vondra.me>
2026-04-03 13:53 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-04-23 07:34 ` David Geier <geidav.pg@gmail.com>
2026-04-29 13:45 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 28+ messages in thread
From: David Geier @ 2026-04-23 07:34 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; Tomas Vondra <tomas@vondra.me>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers
Hi!
I finally got around taking a look at this patch.
On 03.04.2026 15:53, KAZAR Ayoub wrote:
>> - I think it would be useful to report how many file descriptors we
>> are allowed to open (it's less than max_files_per_process, depending
>> on the ulimits etc.)
>>
> Agree, This should be max_safe_fds calculated by postmaster, I added this
> but let me know if its acceptable to export max_safe_fds in the way I did.
An alternative to including it in the view would be using a GUC of type
PGC_INTERNAL. That seems more inline with how we expose other PostgreSQL
internal read-only variables that don't change.
Or is there an advantage to including max_safe_fds in the view?
>> The global cache stats is going to be virtually free (at least the
>> hits/misses, I'm not sure about the number of entries and bytes), and
>> it's obviously useful for tuning the max_files_per_process GUC. I'd even
>> contemplate getting this into PG19, maybe.
The number of used entries already exists, see nfile in fd.c.
Including the total cache size would also be virtually free if we don't
iterate over all VFDs each time, but update the size as we go. That
would have to happen when resizing the cache and when populating /
freeing a cache entry because extra memory is allocated / freed for
Vfd::fileName.
I'm happy to code this up if there's agreement that it's sensible to
include it, in the current version of the patch or a follow-up patch.
Beyond that:
While looking through the code I saw a mistake (repetition of "that") in
a comment in existing code. Maybe you want to fix that as well right away?
/*
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
*/
bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
The indentation of the type at the end of the following two structs is
inconsistent with the rest of the files.
typedef struct PgStatShared_VfdCache
{
/* lock protects ->stats */
LWLock lock;
PgStat_VfdCacheStats stats;
} PgStatShared_VfdCache;
typedef struct PgStat_VfdCacheStats
{
PgStat_Counter vfd_hits; /* fd was open, no open() was needed */
PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was required */
TimestampTz stat_reset_timestamp;
} PgStat_VfdCacheStats;
Apart from these nit comments the patch looks good to me.
--
David Geier
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-31 18:27 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Tomas Vondra <tomas@vondra.me>
2026-04-03 13:53 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-04-23 07:34 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
@ 2026-04-29 13:45 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-05-01 13:56 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
0 siblings, 1 reply; 28+ messages in thread
From: KAZAR Ayoub @ 2026-04-29 13:45 UTC (permalink / raw)
To: David Geier <geidav.pg@gmail.com>; +Cc: Tomas Vondra <tomas@vondra.me>; Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers
Hello,
On Thu, Apr 23, 2026, 9:34 AM David Geier <geidav.pg@gmail.com> wrote:
> Hi!
>
> I finally got around taking a look at this patch.
>
> On 03.04.2026 15:53, KAZAR Ayoub wrote:
> >> - I think it would be useful to report how many file descriptors we
> >> are allowed to open (it's less than max_files_per_process, depending
> >> on the ulimits etc.)
> >>
> > Agree, This should be max_safe_fds calculated by postmaster, I added this
> > but let me know if its acceptable to export max_safe_fds in the way I
> did.
>
> An alternative to including it in the view would be using a GUC of type
> PGC_INTERNAL. That seems more inline with how we expose other PostgreSQL
> internal read-only variables that don't change.
>
> Or is there an advantage to including max_safe_fds in the view?
>
There's no significant advantage other than seeing all info related to
vfdcache together in one view
Also in fd.h i just remembered:
/*
* This is private to fd.c, but exported for
save/restore_backend_variables()
*/
extern PGDLLIMPORT int max_safe_fds;
I don't think it's intended to be exported for reads like i'm doing (which
can be fine?), or maybe update this comment about its export.
>
> >> The global cache stats is going to be virtually free (at least the
> >> hits/misses, I'm not sure about the number of entries and bytes), and
> >> it's obviously useful for tuning the max_files_per_process GUC. I'd even
> >> contemplate getting this into PG19, maybe.
>
> The number of used entries already exists, see nfile in fd.c.
>
Would one want the number of all entries (i.e SizeVfdCache see fd.c) or the
number of used entries (i.e entries with fds in use, which is nfile) ? I
thought of the first, that's what 0002 patch contains for the moment.
>
> Including the total cache size would also be virtually free if we don't
> iterate over all VFDs each time, but update the size as we go. That
> would have to happen when resizing the cache and when populating /
> freeing a cache entry because extra memory is allocated / freed for
> Vfd::fileName.
>
Is it a big deal if we miss some bytes of filename globally ?
>
> I'm happy to code this up if there's agreement that it's sensible to
> include it, in the current version of the patch or a follow-up patch.
>
> Beyond that:
>
> While looking through the code I saw a mistake (repetition of "that") in
> a comment in existing code. Maybe you want to fix that as well right away?
Noted.
>
> /*
> * For variable-numbered stats: flush pending stats. Required if
> pending
> * data is used. See flush_static_cb when dealing with stats data
> that
> * that cannot use PgStat_EntryRef->pending.
> */
> bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool
> nowait);
>
> The indentation of the type at the end of the following two structs is
> inconsistent with the rest of the files.
>
Fixed.
That's pg_indent doing me dirty, although i know it's wrong i didn't
understand why it kept indenting like this, only in those two structs.
>
> typedef struct PgStatShared_VfdCache
> {
> /* lock protects ->stats */
> LWLock lock;
> PgStat_VfdCacheStats stats;
> } PgStatShared_VfdCache;
>
> typedef struct PgStat_VfdCacheStats
> {
> PgStat_Counter vfd_hits; /* fd was open, no open() was
> needed */
> PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was
> required */
> TimestampTz stat_reset_timestamp;
> } PgStat_VfdCacheStats;
>
> Apart from these nit comments the patch looks good to me.
>
Thanks for the review!
Other than the above small changes, i'll be moving forward with 0002 which
is also ready.
Regards,
Ayoub
Attachments:
[text/x-patch] v4-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch (24.4K, ../../CA+K2Ruk55=2fBftAMg3Y=--+6uSNF05UVmu8w8S8FdJ+ektQcg@mail.gmail.com/3-v4-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch)
download | inline diff:
From a4de812dbd99474ecb18654331e6516fff740452 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Wed, 29 Apr 2026 14:39:18 +0200
Subject: [PATCH] Add pg_stat_vfdcache view for VFD cache statistics
PostgreSQL's virtual file descriptor (VFD) layer maintains a
per-backend cache of open file descriptors bounded by
max_files_per_process (default 1000). When the cache is full, the
least-recently-used entry is evicted (its OS fd closed) so a new file
can be opened. A subsequent access to an evicted file must call
open() again.
A trivial example is with partitioned tables: a table with 1500
partitions requires up to many file descriptors per full scan (main
fork, vm ...), which is more than the default limit, causing
potential evictions and reopens.
This commit adds:
pg_stat_vfdcache -- a single-row view exposing cluster-wide VFD cache
statistics:
hits number of VFD cache hits
misses number of VFD cache misses
max_open_fds maximum number of file descriptors available to each backend process
hit_ratio hits / (hits + misses)
stats_reset timestamp of last counter reset
pg_stat_reset_vfdcache() -- resets shared VFD counters
The implementation follows the same cumulative shared statistics infrastructure like pgstat_bgwriter and others do.
Event counting remains cheap in backend-local pending storage and is flushed
into shared fixed stats which requires locking.
Hit and miss counters are placed in FileAccess(), which is the
single gate through which all VFD-mediated file reads, writes,
truncations, and size checks pass.
---
doc/src/sgml/monitoring.sgml | 107 ++++++++++++++++++
src/backend/catalog/system_views.sql | 16 +++
src/backend/storage/file/fd.c | 5 +-
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 17 +++
src/backend/utils/activity/pgstat_vfdcache.c | 113 +++++++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 47 +++++++-
src/include/catalog/pg_proc.dat | 30 +++++
src/include/pgstat.h | 26 +++++
src/include/utils/pgstat_internal.h | 22 +++-
src/include/utils/pgstat_kind.h | 3 +-
src/test/regress/expected/rules.out | 8 ++
src/test/regress/expected/stats.out | 16 ++-
src/test/regress/sql/stats.sql | 5 +
15 files changed, 412 insertions(+), 5 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_vfdcache.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 08d5b824552..fd484de0d47 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -543,6 +543,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_vfdcache</structname><indexterm><primary>pg_stat_vfdcache</primary></indexterm></entry>
+ <entry>One row only, showing cluster-wide statistics about virtual file
+ descriptor (VFD) cache activity. See
+ <link linkend="monitoring-pg-stat-vfdcache-view">
+ <structname>pg_stat_vfdcache</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_wal</structname><indexterm><primary>pg_stat_wal</primary></indexterm></entry>
<entry>One row only, showing statistics about WAL activity. See
@@ -3613,6 +3622,89 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
</sect2>
+ <sect2 id="monitoring-pg-stat-vfdcache-view">
+ <title><structname>pg_stat_vfdcache</structname></title>
+
+ <indexterm zone="monitoring-pg-stat-vfdcache-view">
+ <primary>pg_stat_vfdcache</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_vfdcache</structname> view will always have a
+ single row, containing data about cluster-wide VFD (Virtual File
+ Descriptor) cache activity.
+ </para>
+
+ <table id="pg-stat-vfdcache-view" xreflabel="pg_stat_vfdcache">
+ <title><structname>pg_stat_vfdcache</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hits</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of file accesses where the physical file descriptor was
+ already open in the cache, requiring no system call
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>misses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of file accesses where the physical file descriptor had
+ been evicted from the cache, requiring <function>open()</function>
+ to be called again before the access could proceed
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>max_open_fds</structfield> <type>integer</type>
+ </para>
+ <para>
+ Maximum number of file descriptors available to each backend process
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hit_ratio</structfield> <type>float8</type>
+ </para>
+ <para>
+ Fraction of file accesses that were cache hits:
+ <literal>hits / (hits + misses)</literal>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which the counters were last reset by
+ <function>pg_stat_reset_vfdcache()</function>
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-pg-stat-wal-view">
<title><structname>pg_stat_wal</structname></title>
@@ -5824,6 +5916,21 @@ description | Waiting for a newly initialized WAL file to reach durable storage
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_stat_reset_vfdcache</primary>
+ </indexterm>
+ <function>pg_stat_reset_vfdcache</function> ()
+ <returnvalue>void</returnvalue>
+ </para>
+ <para>
+ Reset shared VFD cache statistics counters to zero. The reset
+ timestamp is recorded in
+ <structname>pg_stat_vfdcache</structname>.<structfield>stats_reset</structfield>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 73a1c1c4670..e79de913538 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1556,3 +1556,19 @@ CREATE VIEW pg_aios AS
SELECT * FROM pg_get_aios();
REVOKE ALL ON pg_aios FROM PUBLIC;
GRANT SELECT ON pg_aios TO pg_read_all_stats;
+
+CREATE VIEW pg_stat_vfdcache AS
+ SELECT
+ pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_max_open_fds() AS max_open_fds,
+ CASE
+ WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
+ THEN NULL::float8
+ ELSE pg_stat_get_vfd_hits()::float8
+ / (pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses())
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
+
+REVOKE ALL ON pg_stat_vfdcache FROM PUBLIC;
+GRANT SELECT ON pg_stat_vfdcache TO PUBLIC;
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index a8be066afe0..6393407864c 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1480,6 +1480,7 @@ static int
FileAccess(File file)
{
int returnValue;
+ bool is_open;
DO_DB(elog(LOG, "FileAccess %d (%s)",
file, VfdCache[file].fileName));
@@ -1488,8 +1489,10 @@ FileAccess(File file)
* Is the file open? If not, open it and put it at the head of the LRU
* ring (possibly closing the least recently used file to get an FD).
*/
+ is_open = !FileIsNotOpen(file);
+ pgstat_count_vfd_access(is_open);
- if (FileIsNotOpen(file))
+ if (!is_open)
{
returnValue = LruInsert(file);
if (returnValue != 0)
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index ca3ef89bf59..fe8fc00d966 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -32,6 +32,7 @@ OBJS = \
pgstat_shmem.o \
pgstat_slru.o \
pgstat_subscription.o \
+ pgstat_vfdcache.o \
pgstat_wal.o \
pgstat_xact.o \
wait_event.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 1aa7ece5290..5c1bcec7f8b 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -17,6 +17,7 @@ backend_sources += files(
'pgstat_shmem.c',
'pgstat_slru.c',
'pgstat_subscription.c',
+ 'pgstat_vfdcache.c',
'pgstat_wal.c',
'pgstat_xact.c',
)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index b67da88c7dc..a64266c1556 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -500,6 +500,23 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
+
+ [PGSTAT_KIND_VFDCACHE] = {
+ .name = "vfdcache",
+
+ .fixed_amount = true,
+ .write_to_file = true,
+
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, vfdcache),
+ .shared_ctl_off = offsetof(PgStat_ShmemControl, vfdcache),
+ .shared_data_off = offsetof(PgStatShared_VfdCache, stats),
+ .shared_data_len = sizeof(((PgStatShared_VfdCache *) 0)->stats),
+
+ .flush_static_cb = pgstat_vfdcache_flush_cb,
+ .init_shmem_cb = pgstat_vfdcache_init_shmem_cb,
+ .reset_all_cb = pgstat_vfdcache_reset_all_cb,
+ .snapshot_cb = pgstat_vfdcache_snapshot_cb,
+ },
};
/*
diff --git a/src/backend/utils/activity/pgstat_vfdcache.c b/src/backend/utils/activity/pgstat_vfdcache.c
new file mode 100644
index 00000000000..b036759031c
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_vfdcache.c
@@ -0,0 +1,113 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_vfdcache.c
+ * Implementation of VFD cache statistics.
+ *
+ * VFD events are first counted in backend-local pending storage and then
+ * flushed into shared-memory cumulative stats, following the same model as
+ * other fixed stats kinds.
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_vfdcache.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "pgstat.h"
+#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+
+/*
+ * Backend-local VFD counters waiting to be flushed.
+ */
+PgStat_VfdCacheStats PendingVfdCacheStats = {0};
+
+/*
+ * Count a VFD cache access as either a hit or miss.
+ */
+void
+pgstat_count_vfd_access(bool hit)
+{
+ if (hit)
+ PendingVfdCacheStats.vfd_hits++;
+ else
+ PendingVfdCacheStats.vfd_misses++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Flush out backend-local pending VFD cache stats.
+ */
+bool
+pgstat_vfdcache_flush_cb(bool nowait)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ if (pg_memory_is_all_zeros(&PendingVfdCacheStats,
+ sizeof(struct PgStat_VfdCacheStats)))
+ return false;
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ stats_shmem->stats.vfd_hits += PendingVfdCacheStats.vfd_hits;
+ stats_shmem->stats.vfd_misses += PendingVfdCacheStats.vfd_misses;
+
+ LWLockRelease(&stats_shmem->lock);
+
+ MemSet(&PendingVfdCacheStats, 0, sizeof(PendingVfdCacheStats));
+
+ return false;
+}
+
+/*
+ * Support function for SQL-callable pg_stat_get_vfd_* functions.
+ */
+PgStat_VfdCacheStats *
+pgstat_fetch_stat_vfdcache(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_VFDCACHE);
+
+ return &pgStatLocal.snapshot.vfdcache;
+}
+
+void
+pgstat_vfdcache_init_shmem_cb(void *stats)
+{
+ PgStatShared_VfdCache *stats_shmem = (PgStatShared_VfdCache *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+void
+pgstat_vfdcache_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ MemSet(&stats_shmem->stats, 0, sizeof(stats_shmem->stats));
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_vfdcache_snapshot_cb(void)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&pgStatLocal.snapshot.vfdcache, &stats_shmem->stats,
+ sizeof(pgStatLocal.snapshot.vfdcache));
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_reset_vfdcache(void)
+{
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 1408de387ea..f35513d7081 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -26,6 +26,7 @@
#include "pgstat.h"
#include "postmaster/bgworker.h"
#include "replication/logicallauncher.h"
+#include "storage/fd.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/acl.h"
@@ -1336,6 +1337,47 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+Datum
+pg_stat_get_vfd_hits(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_hits);
+}
+
+Datum
+pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
+}
+
+Datum
+pg_stat_get_vfd_max_open_fds(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT32(max_safe_fds);
+}
+
+Datum
+pg_stat_get_vfd_stat_reset_time(PG_FUNCTION_ARGS)
+{
+ TimestampTz ts = pgstat_fetch_stat_vfdcache()->stat_reset_timestamp;
+
+ if (ts == 0)
+ PG_RETURN_NULL();
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+}
+
+/*
+ * pg_stat_reset_vfdcache
+ * Reset shared VFD cache counters.
+ */
+Datum
+pg_stat_reset_vfdcache(PG_FUNCTION_ARGS)
+{
+ pgstat_reset_vfdcache();
+ PG_RETURN_VOID();
+}
+
+
/*
* When adding a new column to the pg_stat_io view and the
* pg_stat_get_backend_io() function, add a new enum value here above
@@ -1961,6 +2003,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
PG_RETURN_VOID();
}
@@ -1981,13 +2024,15 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
else if (strcmp(target, "slru") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
+ else if (strcmp(target, "vfdcache") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
else if (strcmp(target, "wal") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"recovery_prefetch\", \"slru\", \"vfdcache\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fa9ae79082b..43132551849 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12768,5 +12768,35 @@
{ oid => '8281', descr => 'hash',
proname => 'hashoid8extended', prorettype => 'int8',
proargtypes => 'oid8 int8', prosrc => 'hashoid8extended' },
+{ oid => '9560',
+ descr => 'statistics: number of VFD cache hits',
+ proname => 'pg_stat_get_vfd_hits',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_hits' },
+{ oid => '9561',
+ descr => 'statistics: number of VFD cache misses',
+ proname => 'pg_stat_get_vfd_misses',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_misses' },
+{ oid => '9564',
+ descr => 'statistics: timestamp of last VFD cache stats reset',
+ proname => 'pg_stat_get_vfd_stat_reset_time',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'timestamptz', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_stat_reset_time' },
+{ oid => '9565',
+ descr => 'statistics: reset shared VFD cache counters',
+ proname => 'pg_stat_reset_vfdcache',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'void', proargtypes => '',
+ prosrc => 'pg_stat_reset_vfdcache' },
+{ oid => '9566',
+ descr => 'statistics: max number of file descriptors available to backend',
+ proname => 'pg_stat_get_vfd_max_open_fds',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int4', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_max_open_fds' },
]
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index dfa2e837638..f2223a538fb 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -272,6 +272,19 @@ typedef struct PgStat_CheckpointerStats
TimestampTz stat_reset_timestamp;
} PgStat_CheckpointerStats;
+/* ---------
+ * PgStat_VfdCacheStats Virtual File Descriptor cache statistics
+ *
+ * Tracks hit/miss events in the VFD cache (fd.c). These counters
+ * are accumulated in shared fixed stats and exposed by pg_stat_vfdcache.
+ * ---------
+ */
+typedef struct PgStat_VfdCacheStats
+{
+ PgStat_Counter vfd_hits; /* fd was open, no open() was needed */
+ PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was required */
+ TimestampTz stat_reset_timestamp;
+} PgStat_VfdCacheStats;
/*
* Types related to counting IO operations
@@ -607,6 +620,13 @@ extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void);
extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern PgStat_VfdCacheStats *pgstat_fetch_stat_vfdcache(void);
+extern void pgstat_reset_vfdcache(void);
+extern void pgstat_count_vfd_access(bool hit);
/*
* Functions in pgstat_io.c
@@ -848,6 +868,12 @@ extern PGDLLIMPORT int pgstat_fetch_consistency;
/* updated directly by bgwriter and bufmgr */
extern PGDLLIMPORT PgStat_BgWriterStats PendingBgWriterStats;
+/*
+ * Variables in pgstat_vfdcache.c
+ */
+
+/* updated by VFD counting functions called from fd.c */
+extern PGDLLIMPORT PgStat_VfdCacheStats PendingVfdCacheStats;
/*
* Variables in pgstat_checkpointer.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index fe463faaf63..5ede0ee255e 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -296,7 +296,7 @@ typedef struct PgStat_KindInfo
/*
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
- * that cannot use PgStat_EntryRef->pending.
+ * cannot use PgStat_EntryRef->pending.
*/
bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
@@ -485,6 +485,13 @@ typedef struct PgStatShared_Wal
PgStat_WalStats stats;
} PgStatShared_Wal;
+typedef struct PgStatShared_VfdCache
+{
+ /* lock protects ->stats */
+ LWLock lock;
+ PgStat_VfdCacheStats stats;
+} PgStatShared_VfdCache;
+
/* ----------
@@ -580,6 +587,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Lock lock;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+ PgStatShared_VfdCache vfdcache;
/*
* Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
@@ -616,6 +624,8 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ PgStat_VfdCacheStats vfdcache;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -730,6 +740,16 @@ extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern bool pgstat_vfdcache_flush_cb(bool nowait);
+extern void pgstat_vfdcache_init_shmem_cb(void *stats);
+extern void pgstat_vfdcache_reset_all_cb(TimestampTz ts);
+extern void pgstat_vfdcache_snapshot_cb(void);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 2d78a029683..319ec0bd63b 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -39,9 +39,10 @@
#define PGSTAT_KIND_LOCK 11
#define PGSTAT_KIND_SLRU 12
#define PGSTAT_KIND_WAL 13
+#define PGSTAT_KIND_VFDCACHE 14
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
+#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_VFDCACHE
#define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1)
/* Custom stats kinds */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index a65a5bf0c4f..034b85a29f2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2415,6 +2415,14 @@ pg_stat_user_tables| SELECT relid,
stats_reset
FROM pg_stat_all_tables
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
+pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_max_open_fds() AS max_open_fds,
+ CASE
+ WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
+ ELSE ((pg_stat_get_vfd_hits())::double precision / ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()))::double precision)
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index e5dcb85abd8..04daadca909 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1161,10 +1161,24 @@ SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
t
(1 row)
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+ ?column?
+----------
+ t
+(1 row)
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
ERROR: unrecognized reset target: "unknown"
-HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", or "wal".
+HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "recovery_prefetch", "slru", "vfdcache", or "wal".
-- Test that reset works for pg_stat_database and pg_stat_database_conflicts
-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so that we
-- have a baseline for comparison. The same for pg_stat_database_conflicts as it shares
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 610fd21fae4..c9096166cc8 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -528,6 +528,11 @@ SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset
SELECT pg_stat_reset_shared('wal');
SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
--
2.34.1
[text/x-patch] v4-0002-Add-VFD-cache-footprint-metrics-to-pg_stat_vfdcache.patch (13.2K, ../../CA+K2Ruk55=2fBftAMg3Y=--+6uSNF05UVmu8w8S8FdJ+ektQcg@mail.gmail.com/4-v4-0002-Add-VFD-cache-footprint-metrics-to-pg_stat_vfdcache.patch)
download | inline diff:
From 16e620af6e1e7669c19efbc240958b25fb2b35db Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Wed, 29 Apr 2026 15:02:59 +0200
Subject: [PATCH 2/2] Add VFD cache footprint metrics to pg_stat_vfdcache
Extend pg_stat_vfdcache with two additional columns:
cache_entries total allocated VFD entries across active backends
cache_bytes total memory used by those entries, in bytes
Unlike hits and misses (cumulative counters), these are live gauges.
Each backend publishes its current VFD cache footprint into backend shared
stats during backend stats flush, and SQL accessors sum those backend shared
stats entries to produce cluster-wide totals.
---
doc/src/sgml/monitoring.sgml | 20 ++++++
src/backend/catalog/system_views.sql | 2 +
src/backend/storage/file/fd.c | 22 +++++++
src/backend/utils/activity/pgstat_backend.c | 48 ++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 71 +++++++++++++++++++++
src/include/catalog/pg_proc.dat | 12 ++++
src/include/pgstat.h | 11 ++++
src/include/storage/fd.h | 2 +
src/include/utils/pgstat_internal.h | 5 +-
src/test/regress/expected/rules.out | 2 +
10 files changed, 194 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index fd484de0d47..2c9881982a9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3671,6 +3671,26 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_entries</structfield> <type>integer</type>
+ </para>
+ <para>
+ Sum of currently allocated VFD cache entries across all active
+ backends
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Sum of memory used by allocated VFD cache entries across all active
+ backends, in bytes
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>max_open_fds</structfield> <type>integer</type>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index e79de913538..3e123736d6d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1561,6 +1561,8 @@ CREATE VIEW pg_stat_vfdcache AS
SELECT
pg_stat_get_vfd_hits() AS hits,
pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_cache_entries() AS cache_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
pg_stat_get_vfd_max_open_fds() AS max_open_fds,
CASE
WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 6393407864c..29fe2c13c07 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1512,6 +1512,28 @@ FileAccess(File file)
return 0;
}
+/*
+ * Return the number of currently allocated VFD entries for this backend,
+ * excluding slot 0 which is used as freelist/LRU header.
+ */
+uint64
+GetVfdCacheEntries(void)
+{
+ if (SizeVfdCache == 0)
+ return 0;
+
+ return (uint64) (SizeVfdCache - 1);
+}
+
+/*
+ * Return the memory footprint of currently allocated VFD entries.
+ */
+uint64
+GetVfdCacheBytes(void)
+{
+ return GetVfdCacheEntries() * sizeof(Vfd);
+}
+
/*
* Called whenever a temporary file is deleted to report its size.
*/
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 73461c9bca5..2345cbbcc49 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -27,6 +27,7 @@
#include "access/xlog.h"
#include "executor/instrument.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/memutils.h"
@@ -48,6 +49,11 @@ static bool backend_has_iostats = false;
*/
static WalUsage prevBackendWalUsage;
+/*
+ * Last per-backend VFD cache gauges published to backend stats.
+ */
+static PgStat_BackendVfdCacheStats prevBackendVfdCacheStats;
+
/*
* Utility routines to report I/O stats for backends, kept here to avoid
* exposing PendingBackendStats to the outside world.
@@ -219,6 +225,19 @@ pgstat_backend_wal_have_pending(void)
return (pgWalUsage.wal_records != prevBackendWalUsage.wal_records);
}
+/*
+ * To determine whether VFD cache gauges changed.
+ */
+static inline bool
+pgstat_backend_vfdcache_have_pending(void)
+{
+ PgStat_Counter current_entries = (PgStat_Counter) GetVfdCacheEntries();
+ PgStat_Counter current_cache_bytes = (PgStat_Counter) GetVfdCacheBytes();
+
+ return current_entries != prevBackendVfdCacheStats.vfd_entries ||
+ current_cache_bytes != prevBackendVfdCacheStats.vfd_cache_bytes;
+}
+
/*
* Flush out locally pending backend WAL statistics. Locking is managed
* by the caller.
@@ -262,6 +281,26 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
prevBackendWalUsage = pgWalUsage;
}
+/*
+ * Flush out locally pending backend VFD cache gauges. Locking is managed
+ * by the caller.
+ * No need to check whether there is pending data here because caller will have done that before acquiring the lock.
+ */
+static void
+pgstat_flush_backend_entry_vfdcache(PgStat_EntryRef *entry_ref)
+{
+ PgStatShared_Backend *shbackendent;
+ PgStat_BackendVfdCacheStats *bktype_shstats;
+
+ shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
+ bktype_shstats = &shbackendent->stats.vfdcache_stats;
+
+ bktype_shstats->vfd_entries = (PgStat_Counter) GetVfdCacheEntries();
+ bktype_shstats->vfd_cache_bytes = (PgStat_Counter) GetVfdCacheBytes();
+
+ prevBackendVfdCacheStats = *bktype_shstats;
+}
+
/*
* Flush out locally pending backend statistics
*
@@ -286,6 +325,11 @@ pgstat_flush_backend(bool nowait, uint32 flags)
pgstat_backend_wal_have_pending())
has_pending_data = true;
+ /* Some VFD cache data pending? */
+ if ((flags & PGSTAT_BACKEND_FLUSH_VFDCACHE) &&
+ pgstat_backend_vfdcache_have_pending())
+ has_pending_data = true;
+
if (!has_pending_data)
return false;
@@ -301,6 +345,9 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_WAL)
pgstat_flush_backend_entry_wal(entry_ref);
+ if (flags & PGSTAT_BACKEND_FLUSH_VFDCACHE)
+ pgstat_flush_backend_entry_vfdcache(entry_ref);
+
pgstat_unlock_entry(entry_ref);
return false;
@@ -339,6 +386,7 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
+ MemSet(&prevBackendVfdCacheStats, 0, sizeof(prevBackendVfdCacheStats));
/*
* Initialize prevBackendWalUsage with pgWalUsage so that
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index f35513d7081..ed19b9d137e 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "access/xact.h"
#include "access/htup_details.h"
#include "access/xlog.h"
#include "access/xlogprefetcher.h"
@@ -1349,6 +1350,76 @@ pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
}
+/*
+ * Sum per-backend VFD gauges across currently active backends.
+ */
+static void
+pgstat_get_vfd_backend_sums(PgStat_Counter *entries_sum,
+ PgStat_Counter *bytes_sum)
+{
+ static TimestampTz cached_stmt_start_ts = 0;
+ static PgStat_Counter cached_entries_sum = 0;
+ static PgStat_Counter cached_bytes_sum = 0;
+ TimestampTz stmt_start_ts = GetCurrentStatementStartTimestamp();
+ int num_backends = pgstat_fetch_stat_numbackends();
+
+ if (cached_stmt_start_ts == stmt_start_ts)
+ {
+ *entries_sum = cached_entries_sum;
+ *bytes_sum = cached_bytes_sum;
+ return;
+ }
+
+ *entries_sum = 0;
+ *bytes_sum = 0;
+
+ for (int curr_backend = 1; curr_backend <= num_backends; curr_backend++)
+ {
+ LocalPgBackendStatus *local_beentry;
+ PgBackendStatus *beentry;
+ PgStat_Backend *backend_stats;
+
+ local_beentry = pgstat_get_local_beentry_by_index(curr_backend);
+ beentry = &local_beentry->backendStatus;
+
+ if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
+ continue;
+
+ backend_stats = pgstat_fetch_stat_backend(local_beentry->proc_number);
+ if (!backend_stats)
+ continue;
+
+ *entries_sum += backend_stats->vfdcache_stats.vfd_entries;
+ *bytes_sum += backend_stats->vfdcache_stats.vfd_cache_bytes;
+ }
+
+ cached_entries_sum = *entries_sum;
+ cached_bytes_sum = *bytes_sum;
+ cached_stmt_start_ts = stmt_start_ts;
+}
+
+Datum
+pg_stat_get_vfd_cache_entries(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter entries_sum;
+ PgStat_Counter bytes_sum;
+
+ pgstat_get_vfd_backend_sums(&entries_sum, &bytes_sum);
+
+ PG_RETURN_INT32((int32) entries_sum);
+}
+
+Datum
+pg_stat_get_vfd_cache_bytes(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter entries_sum;
+ PgStat_Counter bytes_sum;
+
+ pgstat_get_vfd_backend_sums(&entries_sum, &bytes_sum);
+
+ PG_RETURN_INT64(bytes_sum);
+}
+
Datum
pg_stat_get_vfd_max_open_fds(PG_FUNCTION_ARGS)
{
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 43132551849..b1d0c55e450 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12780,6 +12780,18 @@
provolatile => 'v', proparallel => 'r',
prorettype => 'int8', proargtypes => '',
prosrc => 'pg_stat_get_vfd_misses' },
+{ oid => '9562',
+ descr => 'statistics: total number of allocated VFD cache entries',
+ proname => 'pg_stat_get_vfd_cache_entries',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int4', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_entries' },
+{ oid => '9563',
+ descr => 'statistics: total memory footprint of VFD cache entries in bytes',
+ proname => 'pg_stat_get_vfd_cache_bytes',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_bytes' },
{ oid => '9564',
descr => 'statistics: timestamp of last VFD cache stats reset',
proname => 'pg_stat_get_vfd_stat_reset_time',
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f2223a538fb..c9a2f00a4b7 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -286,6 +286,16 @@ typedef struct PgStat_VfdCacheStats
TimestampTz stat_reset_timestamp;
} PgStat_VfdCacheStats;
+/* -------
+ * PgStat_BackendVfdCacheStats Per-backend VFD cache gauges
+ * -------
+ */
+typedef struct PgStat_BackendVfdCacheStats
+{
+ PgStat_Counter vfd_entries;
+ PgStat_Counter vfd_cache_bytes;
+} PgStat_BackendVfdCacheStats;
+
/*
* Types related to counting IO operations
*/
@@ -536,6 +546,7 @@ typedef struct PgStat_Backend
TimestampTz stat_reset_timestamp;
PgStat_BktypeIO io_stats;
PgStat_WalCounters wal_counters;
+ PgStat_BackendVfdCacheStats vfdcache_stats;
} PgStat_Backend;
/* ---------
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 8ac466fd346..17da82c915b 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -149,6 +149,8 @@ extern char *FilePathName(File file);
extern int FileGetRawDesc(File file);
extern int FileGetRawFlags(File file);
extern mode_t FileGetRawMode(File file);
+extern uint64 GetVfdCacheEntries(void);
+extern uint64 GetVfdCacheBytes(void);
/* Operations used for sharing named temporary files */
extern File PathNameCreateTemporaryFile(const char *path, bool error_on_failure);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 5ede0ee255e..5f830c107fc 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -715,7 +715,10 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
+#define PGSTAT_BACKEND_FLUSH_VFDCACHE (1 << 2) /* Flush VFD cache gauges */
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | \
+ PGSTAT_BACKEND_FLUSH_WAL | \
+ PGSTAT_BACKEND_FLUSH_VFDCACHE)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 034b85a29f2..f424eb612e5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2417,6 +2417,8 @@ pg_stat_user_tables| SELECT relid,
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_cache_entries() AS cache_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
pg_stat_get_vfd_max_open_fds() AS max_open_fds,
CASE
WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
--
2.34.1
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-31 18:27 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Tomas Vondra <tomas@vondra.me>
2026-04-03 13:53 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-04-23 07:34 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-04-29 13:45 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-05-01 13:56 ` David Geier <geidav.pg@gmail.com>
2026-07-12 22:44 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Ayoub Kazar <kazarayoub2004@gmail.com>
0 siblings, 1 reply; 28+ messages in thread
From: David Geier @ 2026-05-01 13:56 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Tomas Vondra <tomas@vondra.me>; Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers
On 29.04.2026 15:45, KAZAR Ayoub wrote:
>>>> The global cache stats is going to be virtually free (at least the
>>>> hits/misses, I'm not sure about the number of entries and bytes), and
>>>> it's obviously useful for tuning the max_files_per_process GUC. I'd even
>>>> contemplate getting this into PG19, maybe.
>>
>> The number of used entries already exists, see nfile in fd.c.
>>
> Would one want the number of all entries (i.e SizeVfdCache see fd.c) or the
> number of used entries (i.e entries with fds in use, which is nfile) ? I
> thought of the first, that's what 0002 patch contains for the moment.
I thought we would expose both. That way we can assess in the field if
being able to shrink the cache would be useful.
>> Including the total cache size would also be virtually free if we don't
>> iterate over all VFDs each time, but update the size as we go. That
>> would have to happen when resizing the cache and when populating /
>> freeing a cache entry because extra memory is allocated / freed for
>> Vfd::fileName.
>>
> Is it a big deal if we miss some bytes of filename globally ?
It's not just some bytes. sizeof(struct vfd) is 56 bytes and fileName
looks typically something like:
- base/5/1249
- pg_wal/000000010000000000000001
- pg_wal/archive_status/000000010000000000000001.ready
- pg_xact/0000
- pg_multixact/offsets/0000
File names can vary in length between 10 - 55 bytes, give or take. Most
files will be table and index segments and WAL files. We could maybe add
a fixed constant as "assumed average file name length" but I'm worried
that might end up being quite wrong.
--
David Geier
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: Add pg_stat_vfdcache view for VFD cache statistics
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-31 18:27 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Tomas Vondra <tomas@vondra.me>
2026-04-03 13:53 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-04-23 07:34 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-04-29 13:45 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-05-01 13:56 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
@ 2026-07-12 22:44 ` Ayoub Kazar <kazarayoub2004@gmail.com>
0 siblings, 0 replies; 28+ messages in thread
From: Ayoub Kazar @ 2026-07-12 22:44 UTC (permalink / raw)
To: David Geier <geidav.pg@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Tomas Vondra <tomas@vondra.me>; Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers
Hello everyone,
This is Ayoub again, I've just changed my email address.
On Sun, Jul 12, 2026 at 11:57 PM David Geier <geidav.pg@gmail.com> wrote:
> On 29.04.2026 15:45, KAZAR Ayoub wrote:
> >>>> The global cache stats is going to be virtually free (at least the
> >>>> hits/misses, I'm not sure about the number of entries and bytes), and
> >>>> it's obviously useful for tuning the max_files_per_process GUC. I'd
> even
> >>>> contemplate getting this into PG19, maybe.
> >>
> >> The number of used entries already exists, see nfile in fd.c.
> >>
> > Would one want the number of all entries (i.e SizeVfdCache see fd.c) or
> the
> > number of used entries (i.e entries with fds in use, which is nfile) ? I
> > thought of the first, that's what 0002 patch contains for the moment.
>
> I thought we would expose both. That way we can assess in the field if
> being able to shrink the cache would be useful.
>
Done.
>
> >> Including the total cache size would also be virtually free if we don't
> >> iterate over all VFDs each time, but update the size as we go. That
> >> would have to happen when resizing the cache and when populating /
> >> freeing a cache entry because extra memory is allocated / freed for
> >> Vfd::fileName.
> >>
> > Is it a big deal if we miss some bytes of filename globally ?
>
> It's not just some bytes. sizeof(struct vfd) is 56 bytes and fileName
> looks typically something like:
>
> - base/5/1249
> - pg_wal/000000010000000000000001
> - pg_wal/archive_status/000000010000000000000001.ready
> - pg_xact/0000
> - pg_multixact/offsets/0000
>
> File names can vary in length between 10 - 55 bytes, give or take. Most
> files will be table and index segments and WAL files. We could maybe add
> a fixed constant as "assumed average file name length" but I'm worried
> that might end up being quite wrong.
I chose to keep an accurate running count of the memory footprint per
backend by tracking both the sizeof(Vfd) and the exact filename string
lengths.
We add the string length to the total footprint when a file is opened, and
subtract it when the VFD is freed (i suppose this is not "Too much" of a
work done, although it's done in a bit hot place); v5 of the patch is
attached.
>
>
> --
> David Geier
>
> Regards,
Ayoub KAZAR
Attachments:
[text/x-patch] v5-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch (24.8K, ../../CADu+CpTwQXRdoKVSnoVRLp5m0UbA_cAU6s+_Og5=hCwKp0JR2Q@mail.gmail.com/3-v5-0001-Add-pg_stat_vfdcache-view-for-VFD-cache-statistics.patch)
download | inline diff:
From 5683435456a7fc5faccc8b4b442fb2dfed756e1f Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Sat, 4 Jul 2026 19:16:16 +0200
Subject: [PATCH 1/2] Add pg_stat_vfdcache view for VFD cache statistics
PostgreSQL's virtual file descriptor (VFD) layer maintains a
per-backend cache of open file descriptors bounded by
max_files_per_process (default 1000). When the cache is full, the
least-recently-used entry is evicted (its OS fd closed) so a new file
can be opened. A subsequent access to an evicted file must call
open() again.
A trivial example is with partitioned tables: a table with 1500
partitions requires up to many file descriptors per full scan (main
fork, vm ...), which is more than the default limit, causing
potential evictions and reopens.
This commit adds:
pg_stat_vfdcache -- a single-row view exposing cluster-wide VFD cache
statistics:
hits number of VFD cache hits
misses number of VFD cache misses
max_open_fds maximum number of file descriptors available to each backend process
hit_ratio hits / (hits + misses)
stats_reset timestamp of last counter reset
pg_stat_reset_vfdcache() -- resets shared VFD counters
The implementation follows the same cumulative shared statistics
infrastructure like pgstat_bgwriter and others do.
Event counting remains cheap in backend-local pending storage and is
flushed into shared fixed stats which requires locking.
Hit and miss counters are placed in FileAccess(), which is the
single gate through which all VFD-mediated file reads, writes,
truncations, and size checks pass.
---
doc/src/sgml/monitoring.sgml | 107 ++++++++++++++++++
src/backend/catalog/system_views.sql | 16 +++
src/backend/storage/file/fd.c | 5 +-
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 17 +++
src/backend/utils/activity/pgstat_vfdcache.c | 113 +++++++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 47 +++++++-
src/include/catalog/pg_proc.dat | 30 +++++
src/include/pgstat.h | 26 +++++
src/include/utils/pgstat_internal.h | 22 +++-
src/include/utils/pgstat_kind.h | 3 +-
src/test/regress/expected/rules.out | 8 ++
src/test/regress/expected/stats.out | 19 +++-
src/test/regress/sql/stats.sql | 5 +
15 files changed, 414 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_vfdcache.c
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 12b9ee20d4a..67acd370215 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -553,6 +553,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_vfdcache</structname><indexterm><primary>pg_stat_vfdcache</primary></indexterm></entry>
+ <entry>One row only, showing cluster-wide statistics about virtual file
+ descriptor (VFD) cache activity. See
+ <link linkend="monitoring-pg-stat-vfdcache-view">
+ <structname>pg_stat_vfdcache</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_wal</structname><indexterm><primary>pg_stat_wal</primary></indexterm></entry>
<entry>One row only, showing statistics about WAL activity. See
@@ -3744,6 +3753,89 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
</sect2>
+ <sect2 id="monitoring-pg-stat-vfdcache-view">
+ <title><structname>pg_stat_vfdcache</structname></title>
+
+ <indexterm zone="monitoring-pg-stat-vfdcache-view">
+ <primary>pg_stat_vfdcache</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_vfdcache</structname> view will always have a
+ single row, containing data about cluster-wide VFD (Virtual File
+ Descriptor) cache activity.
+ </para>
+
+ <table id="pg-stat-vfdcache-view" xreflabel="pg_stat_vfdcache">
+ <title><structname>pg_stat_vfdcache</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hits</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of file accesses where the physical file descriptor was
+ already open in the cache, requiring no system call
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>misses</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of file accesses where the physical file descriptor had
+ been evicted from the cache, requiring <function>open()</function>
+ to be called again before the access could proceed
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>max_open_fds</structfield> <type>integer</type>
+ </para>
+ <para>
+ Maximum number of file descriptors available to each backend process
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>hit_ratio</structfield> <type>float8</type>
+ </para>
+ <para>
+ Fraction of file accesses that were cache hits:
+ <literal>hits / (hits + misses)</literal>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which the counters were last reset by
+ <function>pg_stat_reset_vfdcache()</function>
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-pg-stat-wal-view">
<title><structname>pg_stat_wal</structname></title>
@@ -5974,6 +6066,21 @@ description | Waiting for a newly initialized WAL file to reach durable storage
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_stat_reset_vfdcache</primary>
+ </indexterm>
+ <function>pg_stat_reset_vfdcache</function> ()
+ <returnvalue>void</returnvalue>
+ </para>
+ <para>
+ Reset shared VFD cache statistics counters to zero. The reset
+ timestamp is recorded in
+ <structname>pg_stat_vfdcache</structname>.<structfield>stats_reset</structfield>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 6c1c5545cb5..42a82ecb07d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1568,3 +1568,19 @@ CREATE VIEW pg_aios AS
SELECT * FROM pg_get_aios();
REVOKE ALL ON pg_aios FROM PUBLIC;
GRANT SELECT ON pg_aios TO pg_read_all_stats;
+
+CREATE VIEW pg_stat_vfdcache AS
+ SELECT
+ pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_max_open_fds() AS max_open_fds,
+ CASE
+ WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
+ THEN NULL::float8
+ ELSE pg_stat_get_vfd_hits()::float8
+ / (pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses())
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
+
+REVOKE ALL ON pg_stat_vfdcache FROM PUBLIC;
+GRANT SELECT ON pg_stat_vfdcache TO PUBLIC;
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 4cf4717f764..4954e639397 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1480,6 +1480,7 @@ static int
FileAccess(File file)
{
int returnValue;
+ bool is_open;
DO_DB(elog(LOG, "FileAccess %d (%s)",
file, VfdCache[file].fileName));
@@ -1488,8 +1489,10 @@ FileAccess(File file)
* Is the file open? If not, open it and put it at the head of the LRU
* ring (possibly closing the least recently used file to get an FD).
*/
+ is_open = !FileIsNotOpen(file);
+ pgstat_count_vfd_access(is_open);
- if (FileIsNotOpen(file))
+ if (!is_open)
{
returnValue = LruInsert(file);
if (returnValue != 0)
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index 5fed953c28a..b14d46998f9 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -33,6 +33,7 @@ OBJS = \
pgstat_shmem.o \
pgstat_slru.o \
pgstat_subscription.o \
+ pgstat_vfdcache.o \
pgstat_wal.o \
pgstat_xact.o \
wait_event.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 470b5dac402..4cd7bb9fb64 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -18,6 +18,7 @@ backend_sources += files(
'pgstat_shmem.c',
'pgstat_slru.c',
'pgstat_subscription.c',
+ 'pgstat_vfdcache.c',
'pgstat_wal.c',
'pgstat_xact.c',
)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index c4fa14f138f..b4c7113381f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -500,6 +500,23 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
+
+ [PGSTAT_KIND_VFDCACHE] = {
+ .name = "vfdcache",
+
+ .fixed_amount = true,
+ .write_to_file = true,
+
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, vfdcache),
+ .shared_ctl_off = offsetof(PgStat_ShmemControl, vfdcache),
+ .shared_data_off = offsetof(PgStatShared_VfdCache, stats),
+ .shared_data_len = sizeof(((PgStatShared_VfdCache *) 0)->stats),
+
+ .flush_static_cb = pgstat_vfdcache_flush_cb,
+ .init_shmem_cb = pgstat_vfdcache_init_shmem_cb,
+ .reset_all_cb = pgstat_vfdcache_reset_all_cb,
+ .snapshot_cb = pgstat_vfdcache_snapshot_cb,
+ },
};
/*
diff --git a/src/backend/utils/activity/pgstat_vfdcache.c b/src/backend/utils/activity/pgstat_vfdcache.c
new file mode 100644
index 00000000000..b036759031c
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_vfdcache.c
@@ -0,0 +1,113 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_vfdcache.c
+ * Implementation of VFD cache statistics.
+ *
+ * VFD events are first counted in backend-local pending storage and then
+ * flushed into shared-memory cumulative stats, following the same model as
+ * other fixed stats kinds.
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_vfdcache.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "pgstat.h"
+#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+
+/*
+ * Backend-local VFD counters waiting to be flushed.
+ */
+PgStat_VfdCacheStats PendingVfdCacheStats = {0};
+
+/*
+ * Count a VFD cache access as either a hit or miss.
+ */
+void
+pgstat_count_vfd_access(bool hit)
+{
+ if (hit)
+ PendingVfdCacheStats.vfd_hits++;
+ else
+ PendingVfdCacheStats.vfd_misses++;
+ pgstat_report_fixed = true;
+}
+
+/*
+ * Flush out backend-local pending VFD cache stats.
+ */
+bool
+pgstat_vfdcache_flush_cb(bool nowait)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ if (pg_memory_is_all_zeros(&PendingVfdCacheStats,
+ sizeof(struct PgStat_VfdCacheStats)))
+ return false;
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ stats_shmem->stats.vfd_hits += PendingVfdCacheStats.vfd_hits;
+ stats_shmem->stats.vfd_misses += PendingVfdCacheStats.vfd_misses;
+
+ LWLockRelease(&stats_shmem->lock);
+
+ MemSet(&PendingVfdCacheStats, 0, sizeof(PendingVfdCacheStats));
+
+ return false;
+}
+
+/*
+ * Support function for SQL-callable pg_stat_get_vfd_* functions.
+ */
+PgStat_VfdCacheStats *
+pgstat_fetch_stat_vfdcache(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_VFDCACHE);
+
+ return &pgStatLocal.snapshot.vfdcache;
+}
+
+void
+pgstat_vfdcache_init_shmem_cb(void *stats)
+{
+ PgStatShared_VfdCache *stats_shmem = (PgStatShared_VfdCache *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+void
+pgstat_vfdcache_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ MemSet(&stats_shmem->stats, 0, sizeof(stats_shmem->stats));
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_vfdcache_snapshot_cb(void)
+{
+ PgStatShared_VfdCache *stats_shmem = &pgStatLocal.shmem->vfdcache;
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&pgStatLocal.snapshot.vfdcache, &stats_shmem->stats,
+ sizeof(pgStatLocal.snapshot.vfdcache));
+ LWLockRelease(&stats_shmem->lock);
+}
+
+void
+pgstat_reset_vfdcache(void)
+{
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 565d0e70768..714918d292c 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -26,6 +26,7 @@
#include "pgstat.h"
#include "postmaster/bgworker.h"
#include "replication/logicallauncher.h"
+#include "storage/fd.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/acl.h"
@@ -1336,6 +1337,47 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+Datum
+pg_stat_get_vfd_hits(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_hits);
+}
+
+Datum
+pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
+}
+
+Datum
+pg_stat_get_vfd_max_open_fds(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT32(max_safe_fds);
+}
+
+Datum
+pg_stat_get_vfd_stat_reset_time(PG_FUNCTION_ARGS)
+{
+ TimestampTz ts = pgstat_fetch_stat_vfdcache()->stat_reset_timestamp;
+
+ if (ts == 0)
+ PG_RETURN_NULL();
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+}
+
+/*
+ * pg_stat_reset_vfdcache
+ * Reset shared VFD cache counters.
+ */
+Datum
+pg_stat_reset_vfdcache(PG_FUNCTION_ARGS)
+{
+ pgstat_reset_vfdcache();
+ PG_RETURN_VOID();
+}
+
+
/*
* When adding a new column to the pg_stat_io view and the
* pg_stat_get_backend_io() function, add a new enum value here above
@@ -2001,6 +2043,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
PG_RETURN_VOID();
}
@@ -2021,13 +2064,15 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
else if (strcmp(target, "slru") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
+ else if (strcmp(target, "vfdcache") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_VFDCACHE);
else if (strcmp(target, "wal") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
- errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"lock\", \"recovery_prefetch\", \"slru\", or \"wal\".")));
+ errhint("Target must be \"archiver\", \"bgwriter\", \"checkpointer\", \"io\", \"lock\", \"recovery_prefetch\", \"slru\", \"vfdcache\", or \"wal\".")));
PG_RETURN_VOID();
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3cb84359adf..52a9579f873 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12717,5 +12717,35 @@
{ oid => '6462', descr => 'hash',
proname => 'hashoid8extended', prorettype => 'int8',
proargtypes => 'oid8 int8', prosrc => 'hashoid8extended' },
+{ oid => '9560',
+ descr => 'statistics: number of VFD cache hits',
+ proname => 'pg_stat_get_vfd_hits',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_hits' },
+{ oid => '9561',
+ descr => 'statistics: number of VFD cache misses',
+ proname => 'pg_stat_get_vfd_misses',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_misses' },
+{ oid => '9564',
+ descr => 'statistics: timestamp of last VFD cache stats reset',
+ proname => 'pg_stat_get_vfd_stat_reset_time',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'timestamptz', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_stat_reset_time' },
+{ oid => '9565',
+ descr => 'statistics: reset shared VFD cache counters',
+ proname => 'pg_stat_reset_vfdcache',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'void', proargtypes => '',
+ prosrc => 'pg_stat_reset_vfdcache' },
+{ oid => '9566',
+ descr => 'statistics: max number of file descriptors available to backend',
+ proname => 'pg_stat_get_vfd_max_open_fds',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int4', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_max_open_fds' },
]
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 58a44857f13..5791894782b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -272,6 +272,19 @@ typedef struct PgStat_CheckpointerStats
TimestampTz stat_reset_timestamp;
} PgStat_CheckpointerStats;
+/* ---------
+ * PgStat_VfdCacheStats Virtual File Descriptor cache statistics
+ *
+ * Tracks hit/miss events in the VFD cache (fd.c). These counters
+ * are accumulated in shared fixed stats and exposed by pg_stat_vfdcache.
+ * ---------
+ */
+typedef struct PgStat_VfdCacheStats
+{
+ PgStat_Counter vfd_hits; /* fd was open, no open() was needed */
+ PgStat_Counter vfd_misses; /* fd was VFD_CLOSED, open() was required */
+ TimestampTz stat_reset_timestamp;
+} PgStat_VfdCacheStats;
/*
* Types related to counting IO operations
@@ -619,6 +632,13 @@ extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void);
extern void pgstat_report_checkpointer(void);
extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern PgStat_VfdCacheStats *pgstat_fetch_stat_vfdcache(void);
+extern void pgstat_reset_vfdcache(void);
+extern void pgstat_count_vfd_access(bool hit);
/*
* Functions in pgstat_io.c
@@ -861,6 +881,12 @@ extern PGDLLIMPORT int pgstat_fetch_consistency;
/* updated directly by bgwriter and bufmgr */
extern PGDLLIMPORT PgStat_BgWriterStats PendingBgWriterStats;
+/*
+ * Variables in pgstat_vfdcache.c
+ */
+
+/* updated by VFD counting functions called from fd.c */
+extern PGDLLIMPORT PgStat_VfdCacheStats PendingVfdCacheStats;
/*
* Variables in pgstat_checkpointer.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index b3dc3ff7d8b..790de4436d9 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -296,7 +296,7 @@ typedef struct PgStat_KindInfo
/*
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
- * that cannot use PgStat_EntryRef->pending.
+ * cannot use PgStat_EntryRef->pending.
*/
bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
@@ -485,6 +485,13 @@ typedef struct PgStatShared_Wal
PgStat_WalStats stats;
} PgStatShared_Wal;
+typedef struct PgStatShared_VfdCache
+{
+ /* lock protects ->stats */
+ LWLock lock;
+ PgStat_VfdCacheStats stats;
+} PgStatShared_VfdCache;
+
/* ----------
@@ -580,6 +587,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_Lock lock;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+ PgStatShared_VfdCache vfdcache;
/*
* Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
@@ -616,6 +624,8 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ PgStat_VfdCacheStats vfdcache;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -731,6 +741,16 @@ extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
+/*
+ * Functions in pgstat_vfdcache.c
+ */
+
+extern bool pgstat_vfdcache_flush_cb(bool nowait);
+extern void pgstat_vfdcache_init_shmem_cb(void *stats);
+extern void pgstat_vfdcache_reset_all_cb(TimestampTz ts);
+extern void pgstat_vfdcache_snapshot_cb(void);
+
+
/*
* Functions in pgstat_database.c
*/
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 2d78a029683..319ec0bd63b 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -39,9 +39,10 @@
#define PGSTAT_KIND_LOCK 11
#define PGSTAT_KIND_SLRU 12
#define PGSTAT_KIND_WAL 13
+#define PGSTAT_KIND_VFDCACHE 14
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
+#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_VFDCACHE
#define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1)
/* Custom stats kinds */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 39905c2de14..d195291a27a 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2423,6 +2423,14 @@ pg_stat_user_tables| SELECT relid,
stats_reset
FROM pg_stat_all_tables
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
+pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
+ pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_max_open_fds() AS max_open_fds,
+ CASE
+ WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
+ ELSE ((pg_stat_get_vfd_hits())::double precision / ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()))::double precision)
+ END AS hit_ratio,
+ pg_stat_get_vfd_stat_reset_time() AS stats_reset;
pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 03cbc1cdef5..8b88fba4540 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -134,7 +134,8 @@ SELECT id, name, fixed_amount,
11 | lock | t | f | t
12 | slru | t | f | t
13 | wal | t | f | t
-(13 rows)
+ 14 | vfdcache | t | f | t
+(14 rows)
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
@@ -1184,10 +1185,24 @@ SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
t
(1 row)
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+ ?column?
+----------
+ t
+(1 row)
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
ERROR: unrecognized reset target: "unknown"
-HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "lock", "recovery_prefetch", "slru", or "wal".
+HINT: Target must be "archiver", "bgwriter", "checkpointer", "io", "lock", "recovery_prefetch", "slru", "vfdcache", or "wal".
-- Test that reset works for pg_stat_database and pg_stat_database_conflicts
-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so that we
-- have a baseline for comparison. The same for pg_stat_database_conflicts as it shares
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 4c265d1245c..1247731c4d1 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -535,6 +535,11 @@ SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset
SELECT pg_stat_reset_shared('wal');
SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal;
+-- Test that reset_shared with vfdcache specified as the stats type works
+SELECT stats_reset AS vfdcache_reset_ts FROM pg_stat_vfdcache \gset
+SELECT pg_stat_reset_shared('vfdcache');
+SELECT stats_reset > :'vfdcache_reset_ts'::timestamptz FROM pg_stat_vfdcache;
+
-- Test error case for reset_shared with unknown stats type
SELECT pg_stat_reset_shared('unknown');
--
2.34.1
[text/x-patch] v5-0002-Add-VFD-cache-footprint-metrics-to-pg_stat_vfdcache.patch (17.6K, ../../CADu+CpTwQXRdoKVSnoVRLp5m0UbA_cAU6s+_Og5=hCwKp0JR2Q@mail.gmail.com/4-v5-0002-Add-VFD-cache-footprint-metrics-to-pg_stat_vfdcache.patch)
download | inline diff:
From 37de112185d8d444f06d55a09837885d7fa24090 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Sat, 4 Jul 2026 19:39:25 +0200
Subject: [PATCH 2/2] Add VFD cache footprint metrics to pg_stat_vfdcache
Extend pg_stat_vfdcache with three additional columns:
open_entries number of currently open file descriptors inside VFD cache
allocated_entries total number of allocated slots in the cache
cache_bytes total memory used by VFD cache, including sizeof(Vfd) for
allocated slots and the heap-allocated fileName strings
These are live gauges. Each backend publishes its current VFD cache
footprint into backend shared stats during backend stats flush, and
SQL accessors sum those backend shared stats entries to produce
cluster-wide totals.
The memory footprint is computed incrementally inside fd.c using a
running counter, avoiding expensive iterations or averages, and
accurately accounting for filename string allocations.
---
doc/src/sgml/monitoring.sgml | 33 ++++++++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/file/fd.c | 48 ++++++++++-
src/backend/utils/activity/pgstat_backend.c | 52 ++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 93 +++++++++++++++++++++
src/include/catalog/pg_proc.dat | 18 ++++
src/include/pgstat.h | 15 ++++
src/include/storage/fd.h | 3 +
src/include/utils/pgstat_internal.h | 6 +-
src/test/regress/expected/rules.out | 3 +
10 files changed, 272 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 67acd370215..1d63d17a2b8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3802,6 +3802,39 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>open_entries</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Sum of VFD slots currently holding an open kernel file descriptor
+ across all active backends (equivalent to summing each backend's
+ <literal>nfile</literal> counter)
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>allocated_entries</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Sum of allocated VFD cache entries across all active backends,
+ including entries whose file descriptor has been closed by the LRU
+ eviction policy but whose slot has not yet been freed
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>cache_bytes</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Sum of memory used by VFD cache entries across all active backends,
+ in bytes; includes <literal>sizeof(Vfd)</literal> for each allocated
+ slot plus the heap-allocated file name string for each populated slot
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>max_open_fds</structfield> <type>integer</type>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 42a82ecb07d..712542d0fdd 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1573,6 +1573,9 @@ CREATE VIEW pg_stat_vfdcache AS
SELECT
pg_stat_get_vfd_hits() AS hits,
pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_cache_open_entries() AS open_entries,
+ pg_stat_get_vfd_cache_allocated_entries() AS allocated_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
pg_stat_get_vfd_max_open_fds() AS max_open_fds,
CASE
WHEN pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses() = 0
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 4954e639397..80f0a695011 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -225,6 +225,14 @@ static Size SizeVfdCache = 0;
*/
static int nfile = 0;
+/*
+ * Running total of memory used by the VFD cache for this backend,
+ * including sizeof(Vfd) for each allocated slot plus the malloc'd
+ * fileName string for each populated slot. Updated incrementally in
+ * AllocateVfd(), PathNameOpenFilePerm(), and FreeVfd().
+ */
+static uint64 VfdCacheFootprintBytes = 0;
+
/*
* Flag to tell whether it's worth scanning VfdCache looking for temp files
* to close
@@ -1444,8 +1452,10 @@ AllocateVfd(void)
VfdCache[0].nextFree = SizeVfdCache;
/*
- * Record the new size
+ * Record the new size and account for the newly allocated Vfd
+ * structs.
*/
+ VfdCacheFootprintBytes += (newCacheSize - SizeVfdCache) * sizeof(Vfd);
SizeVfdCache = newCacheSize;
}
@@ -1466,6 +1476,7 @@ FreeVfd(File file)
if (vfdP->fileName != NULL)
{
+ VfdCacheFootprintBytes -= strlen(vfdP->fileName) + 1;
free(vfdP->fileName);
vfdP->fileName = NULL;
}
@@ -1512,6 +1523,40 @@ FileAccess(File file)
return 0;
}
+/*
+ * Return the number of VFD slots currently holding an open file descriptor.
+ * This is the nfile counter, which tracks FDs actually open in the kernel.
+ */
+uint64
+GetVfdCacheOpenEntries(void)
+{
+ return (uint64) nfile;
+}
+
+/*
+ * Return the number of currently allocated VFD entries for this backend,
+ * excluding slot 0 which is used as freelist/LRU header.
+ */
+uint64
+GetVfdCacheAllocatedEntries(void)
+{
+ if (SizeVfdCache == 0)
+ return 0;
+
+ return (uint64) (SizeVfdCache - 1);
+}
+
+/*
+ * Return the current memory footprint of the VFD cache for this backend.
+ * This includes sizeof(Vfd) for each allocated slot and the malloc'd
+ * fileName bytes for each currently-populated slot.
+ */
+uint64
+GetVfdCacheBytes(void)
+{
+ return VfdCacheFootprintBytes;
+}
+
/*
* Called whenever a temporary file is deleted to report its size.
*/
@@ -1624,6 +1669,7 @@ PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode)
vfdP->fd));
vfdP->fileName = fnamecopy;
+ VfdCacheFootprintBytes += strlen(fnamecopy) + 1;
/* Saved flags are adjusted to be OK for re-opening file */
vfdP->fileFlags = fileFlags & ~(O_CREAT | O_TRUNC | O_EXCL);
vfdP->fileMode = fileMode;
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index b736b2ccc6f..a493b559911 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -27,6 +27,7 @@
#include "access/xlog.h"
#include "executor/instrument.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/memutils.h"
@@ -49,6 +50,12 @@ static bool backend_has_lockstats = false;
*/
static WalUsage prevBackendWalUsage;
+/*
+ * Last per-backend VFD cache gauges published to backend stats.
+ * Used to detect whether anything changed since the last flush.
+ */
+static PgStat_BackendVfdCacheStats prevBackendVfdCacheStats;
+
/*
* Utility routines to report I/O stats for backends, kept here to avoid
* exposing PendingBackendStats to the outside world.
@@ -253,6 +260,21 @@ pgstat_backend_wal_have_pending(void)
return (pgWalUsage.wal_records != prevBackendWalUsage.wal_records);
}
+/*
+ * Determine whether VFD cache gauges have changed since the last flush.
+ */
+static inline bool
+pgstat_backend_vfdcache_have_pending(void)
+{
+ PgStat_Counter cur_open = (PgStat_Counter) GetVfdCacheOpenEntries();
+ PgStat_Counter cur_alloc = (PgStat_Counter) GetVfdCacheAllocatedEntries();
+ PgStat_Counter cur_bytes = (PgStat_Counter) GetVfdCacheBytes();
+
+ return cur_open != prevBackendVfdCacheStats.open_entries ||
+ cur_alloc != prevBackendVfdCacheStats.allocated_entries ||
+ cur_bytes != prevBackendVfdCacheStats.cache_bytes;
+}
+
/*
* Flush out locally pending backend WAL statistics. Locking is managed
* by the caller.
@@ -326,6 +348,27 @@ pgstat_flush_backend_entry_lock(PgStat_EntryRef *entry_ref)
backend_has_lockstats = false;
}
+/*
+ * Flush out locally pending backend VFD cache gauges. Locking is managed
+ * by the caller. No need to check for pending data here; the caller does
+ * that before acquiring the lock.
+ */
+static void
+pgstat_flush_backend_entry_vfdcache(PgStat_EntryRef *entry_ref)
+{
+ PgStatShared_Backend *shbackendent;
+ PgStat_BackendVfdCacheStats *bktype_shstats;
+
+ shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
+ bktype_shstats = &shbackendent->stats.vfdcache_stats;
+
+ bktype_shstats->open_entries = (PgStat_Counter) GetVfdCacheOpenEntries();
+ bktype_shstats->allocated_entries = (PgStat_Counter) GetVfdCacheAllocatedEntries();
+ bktype_shstats->cache_bytes = (PgStat_Counter) GetVfdCacheBytes();
+
+ prevBackendVfdCacheStats = *bktype_shstats;
+}
+
/*
* Flush out locally pending backend statistics
*
@@ -354,6 +397,11 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats)
has_pending_data = true;
+ /* Some VFD cache data pending? */
+ if ((flags & PGSTAT_BACKEND_FLUSH_VFDCACHE) &&
+ pgstat_backend_vfdcache_have_pending())
+ has_pending_data = true;
+
if (!has_pending_data)
return false;
@@ -372,6 +420,9 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_LOCK)
pgstat_flush_backend_entry_lock(entry_ref);
+ if (flags & PGSTAT_BACKEND_FLUSH_VFDCACHE)
+ pgstat_flush_backend_entry_vfdcache(entry_ref);
+
pgstat_unlock_entry(entry_ref);
return false;
@@ -411,6 +462,7 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
backend_has_lockstats = false;
+ MemSet(&prevBackendVfdCacheStats, 0, sizeof(prevBackendVfdCacheStats));
/*
* Initialize prevBackendWalUsage with pgWalUsage so that
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 714918d292c..55835645bae 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/htup_details.h"
+#include "access/xact.h"
#include "access/xlog.h"
#include "access/xlogprefetcher.h"
#include "catalog/catalog.h"
@@ -1349,6 +1350,98 @@ pg_stat_get_vfd_misses(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_vfdcache()->vfd_misses);
}
+/*
+ * Sum per-backend VFD cache gauges across all currently active backends.
+ * Results are cached for the duration of the current SQL statement to
+ * avoid redundant scans when the view calls multiple getter functions.
+ */
+static void
+pgstat_get_vfd_backend_sums(PgStat_Counter *open_sum,
+ PgStat_Counter *alloc_sum,
+ PgStat_Counter *bytes_sum)
+{
+ static TimestampTz cached_stmt_start_ts = 0;
+ static PgStat_Counter cached_open_sum = 0;
+ static PgStat_Counter cached_alloc_sum = 0;
+ static PgStat_Counter cached_bytes_sum = 0;
+ TimestampTz stmt_start_ts = GetCurrentStatementStartTimestamp();
+ int num_backends = pgstat_fetch_stat_numbackends();
+
+ if (cached_stmt_start_ts == stmt_start_ts)
+ {
+ *open_sum = cached_open_sum;
+ *alloc_sum = cached_alloc_sum;
+ *bytes_sum = cached_bytes_sum;
+ return;
+ }
+
+ *open_sum = 0;
+ *alloc_sum = 0;
+ *bytes_sum = 0;
+
+ for (int curr_backend = 1; curr_backend <= num_backends; curr_backend++)
+ {
+ LocalPgBackendStatus *local_beentry;
+ PgBackendStatus *beentry;
+ PgStat_Backend *backend_stats;
+
+ local_beentry = pgstat_get_local_beentry_by_index(curr_backend);
+ beentry = &local_beentry->backendStatus;
+
+ if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
+ continue;
+
+ backend_stats = pgstat_fetch_stat_backend(local_beentry->proc_number);
+ if (!backend_stats)
+ continue;
+
+ *open_sum += backend_stats->vfdcache_stats.open_entries;
+ *alloc_sum += backend_stats->vfdcache_stats.allocated_entries;
+ *bytes_sum += backend_stats->vfdcache_stats.cache_bytes;
+ }
+
+ cached_open_sum = *open_sum;
+ cached_alloc_sum = *alloc_sum;
+ cached_bytes_sum = *bytes_sum;
+ cached_stmt_start_ts = stmt_start_ts;
+}
+
+Datum
+pg_stat_get_vfd_cache_open_entries(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter open_sum;
+ PgStat_Counter alloc_sum;
+ PgStat_Counter bytes_sum;
+
+ pgstat_get_vfd_backend_sums(&open_sum, &alloc_sum, &bytes_sum);
+
+ PG_RETURN_INT64(open_sum);
+}
+
+Datum
+pg_stat_get_vfd_cache_allocated_entries(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter open_sum;
+ PgStat_Counter alloc_sum;
+ PgStat_Counter bytes_sum;
+
+ pgstat_get_vfd_backend_sums(&open_sum, &alloc_sum, &bytes_sum);
+
+ PG_RETURN_INT64(alloc_sum);
+}
+
+Datum
+pg_stat_get_vfd_cache_bytes(PG_FUNCTION_ARGS)
+{
+ PgStat_Counter open_sum;
+ PgStat_Counter alloc_sum;
+ PgStat_Counter bytes_sum;
+
+ pgstat_get_vfd_backend_sums(&open_sum, &alloc_sum, &bytes_sum);
+
+ PG_RETURN_INT64(bytes_sum);
+}
+
Datum
pg_stat_get_vfd_max_open_fds(PG_FUNCTION_ARGS)
{
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 52a9579f873..9435286a171 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12747,5 +12747,23 @@
provolatile => 'v', proparallel => 'r',
prorettype => 'int4', proargtypes => '',
prosrc => 'pg_stat_get_vfd_max_open_fds' },
+{ oid => '9562',
+ descr => 'statistics: number of VFD slots with an open file descriptor across backends',
+ proname => 'pg_stat_get_vfd_cache_open_entries',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_open_entries' },
+{ oid => '9563',
+ descr => 'statistics: total number of allocated VFD cache entries across backends',
+ proname => 'pg_stat_get_vfd_cache_allocated_entries',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_allocated_entries' },
+{ oid => '9567',
+ descr => 'statistics: total memory footprint of VFD cache across backends in bytes',
+ proname => 'pg_stat_get_vfd_cache_bytes',
+ provolatile => 'v', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_vfd_cache_bytes' },
]
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5791894782b..7c7a18c22bb 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -286,6 +286,20 @@ typedef struct PgStat_VfdCacheStats
TimestampTz stat_reset_timestamp;
} PgStat_VfdCacheStats;
+/* -------
+ * PgStat_BackendVfdCacheStats Per-backend VFD cache gauges
+ *
+ * These are live gauges (not cumulative counters) snapshotted from each
+ * backend's fd.c variables during stats flush.
+ * -------
+ */
+typedef struct PgStat_BackendVfdCacheStats
+{
+ PgStat_Counter open_entries; /* FDs currently open (= nfile) */
+ PgStat_Counter allocated_entries; /* total allocated VFD slots */
+ PgStat_Counter cache_bytes; /* memory footprint in bytes */
+} PgStat_BackendVfdCacheStats;
+
/*
* Types related to counting IO operations
*/
@@ -537,6 +551,7 @@ typedef struct PgStat_Backend
PgStat_BktypeIO io_stats;
PgStat_WalCounters wal_counters;
PgStat_PendingLock lock_stats;
+ PgStat_BackendVfdCacheStats vfdcache_stats;
} PgStat_Backend;
/* ---------
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 8ac466fd346..672e18ca19e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -149,6 +149,9 @@ extern char *FilePathName(File file);
extern int FileGetRawDesc(File file);
extern int FileGetRawFlags(File file);
extern mode_t FileGetRawMode(File file);
+extern uint64 GetVfdCacheOpenEntries(void);
+extern uint64 GetVfdCacheAllocatedEntries(void);
+extern uint64 GetVfdCacheBytes(void);
/* Operations used for sharing named temporary files */
extern File PathNameCreateTemporaryFile(const char *path, bool error_on_failure);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 790de4436d9..eb64a96af20 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -716,7 +716,11 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL | PGSTAT_BACKEND_FLUSH_LOCK)
+#define PGSTAT_BACKEND_FLUSH_VFDCACHE (1 << 3) /* Flush VFD cache gauges */
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | \
+ PGSTAT_BACKEND_FLUSH_WAL | \
+ PGSTAT_BACKEND_FLUSH_LOCK | \
+ PGSTAT_BACKEND_FLUSH_VFDCACHE)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d195291a27a..9c9df43bc2f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2425,6 +2425,9 @@ pg_stat_user_tables| SELECT relid,
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
pg_stat_vfdcache| SELECT pg_stat_get_vfd_hits() AS hits,
pg_stat_get_vfd_misses() AS misses,
+ pg_stat_get_vfd_cache_open_entries() AS open_entries,
+ pg_stat_get_vfd_cache_allocated_entries() AS allocated_entries,
+ pg_stat_get_vfd_cache_bytes() AS cache_bytes,
pg_stat_get_vfd_max_open_fds() AS max_open_fds,
CASE
WHEN ((pg_stat_get_vfd_hits() + pg_stat_get_vfd_misses()) = 0) THEN NULL::double precision
--
2.34.1
^ permalink raw reply [nested|flat] 28+ messages in thread
end of thread, other threads:[~2026-07-12 22:44 UTC | newest]
Thread overview: 28+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-07-05 14:24 [PATCH 17/17] User and development documentation. Antonin Houska <ah@cybertec.at>
2024-05-24 02:26 [PATCH v20 7/8] Row pattern recognition patch (tests). Tatsuo Ishii <ishii@postgresql.org>
2026-01-06 11:06 [PATCH v7 4/5] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v9 4/5] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v10 4/5] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v1 2/3] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v4 3/4] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v11 4/5] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v8 4/5] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v3 2/3] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v6 4/5] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v11 4/5] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v2 2/3] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-06 11:06 [PATCH v5 3/4] Remove useless calls to flush some stats Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-21 16:58 Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-23 11:22 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-03-23 12:35 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-23 23:36 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-24 14:09 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-29 18:23 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-29 19:46 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-31 14:54 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-03-31 18:27 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Tomas Vondra <tomas@vondra.me>
2026-04-03 13:53 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-04-23 07:34 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-04-29 13:45 ` Re: Add pg_stat_vfdcache view for VFD cache statistics KAZAR Ayoub <ma_kazar@esi.dz>
2026-05-01 13:56 ` Re: Add pg_stat_vfdcache view for VFD cache statistics David Geier <geidav.pg@gmail.com>
2026-07-12 22:44 ` Re: Add pg_stat_vfdcache view for VFD cache statistics Ayoub Kazar <kazarayoub2004@gmail.com>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox