agora inbox for [email protected]
help / color / mirror / Atom feedsrc/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h
278+ messages / 4 participants
[nested] [flat]
* src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h
@ 2020-05-08 19:16 Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Christoph Berg @ 2020-05-08 19:16 UTC (permalink / raw)
To: pgsql-hackers
I believe check_pg_config as used by src/test/ssl/t/002_scram.pl
shouldn't rely on /usr/include/postgresql/pg_config.h but use the file
from the build tree instead:
src/test/perl/TestLib.pm:
Return the number of matches of the given regular expression
within the installation's C<pg_config.h>.
=cut
sub check_pg_config
{
my ($regexp) = @_;
my ($stdout, $stderr);
my $result = IPC::Run::run [ 'pg_config', '--includedir' ], '>',
\$stdout, '2>', \$stderr
or die "could not execute pg_config";
chomp($stdout);
$stdout =~ s/\r$//;
open my $pg_config_h, '<', "$stdout/pg_config.h" or die "$!"; <-- here
my $match = (grep { /^$regexp/ } <$pg_config_h>);
close $pg_config_h;
return $match;
}
src/test/ssl/README claims that it is possible to run the "ssl" extra
test from make check (as opposed to installcheck).
Christoph
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
@ 2020-05-08 19:34 ` Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Tom Lane @ 2020-05-08 19:34 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; +Cc: pgsql-hackers
Christoph Berg <[email protected]> writes:
> I believe check_pg_config as used by src/test/ssl/t/002_scram.pl
> shouldn't rely on /usr/include/postgresql/pg_config.h but use the file
> from the build tree instead:
But during "make check", that should be executing pg_config from the
the temporary installation, so we should get the right answer no?
Conversely, in "make installcheck" scenarios, we definitely do want
the value from the installed file, or so I should think.
Do you have a concrete scenario where you're getting wrong behavior?
regards, tom lane
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
@ 2020-05-08 19:45 ` Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Christoph Berg @ 2020-05-08 19:45 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Re: Tom Lane
> But during "make check", that should be executing pg_config from the
> the temporary installation, so we should get the right answer no?
>
> Conversely, in "make installcheck" scenarios, we definitely do want
> the value from the installed file, or so I should think.
>
> Do you have a concrete scenario where you're getting wrong behavior?
I just added the extra tests to the postgresql-13 package and got
this:
$ cat build/src/test/ssl/tmp_check/log/regress_log_002_scram
No such file or directory at /srv/projects/postgresql/pg/master/build/../src/test/perl/TestLib.pm line 595.
But I just realized that the problem is caused by a Debian-specific
patch that removes the find_my_exec logic from pg_config and replaces
it with PGBINDIR. We need that patch because we install pg_config to
both /usr/bin and /usr/lib/postgresql/<version>/bin and want the same
output from both.
I'll revisit that to see if we can come up with a different solution.
Christoph
^ permalink raw reply [nested|flat] 278+ messages in thread
* ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
@ 2020-05-13 16:05 ` Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Christoph Berg @ 2020-05-13 16:05 UTC (permalink / raw)
To: pgsql-hackers
> I just added the extra tests to the postgresql-13 package and got
> this:
Some other problem emerged here in the ldap test:
17:28:58
17:28:58 # Failed test 'StartTLS'
17:28:58 # at t/001_auth.pl line 169.
17:28:58 # got: '2'
17:28:58 # expected: '0'
17:28:58
17:28:58 # Failed test 'LDAPS'
17:28:58 # at t/001_auth.pl line 169.
17:28:58 # got: '2'
17:28:58 # expected: '0'
17:28:59
17:28:59 # Failed test 'LDAPS with URL'
17:28:59 # at t/001_auth.pl line 169.
17:28:59 # got: '2'
17:28:59 # expected: '0'
17:28:59 # Looks like you failed 3 tests of 22.
17:28:59 t/001_auth.pl ..
...
17:28:59 # diagnostic message
17:28:59 ok 18 - any attempt fails due to bad search pattern
17:28:59 # TLS
17:28:59 not ok 19 - StartTLS
17:28:59 not ok 20 - LDAPS
17:28:59 not ok 21 - LDAPS with URL
17:28:59 ok 22 - bad combination of LDAPS and StartTLS
17:28:59 Dubious, test returned 3 (wstat 768, 0x300)
src/test/ldap/tmp_check/log/slapd.log is empty.
It consistently fails on the build server, but works on my notebook.
Maybe that simply means slapd is crashing, but there's no slapd
output. Would it be possible to start slapd with "-d 255", even if
that means it doesn't background itself?
That'd be in src/test/ldap/t/001_auth.pl:
system_or_bail $slapd, '-f', $slapd_conf, '-h', "$ldap_url $ldaps_url";
END
{
kill 'INT', `cat $slapd_pidfile` if -f $slapd_pidfile;
}
Server and test output below:
17:28:59 2020-05-13 15:28:58.136 UTC [31564] LOG: starting PostgreSQL 13devel (Debian 13~~devel~20200513.1505-1~801.git043e3e0.pgdg+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 9.3.0-12) 9.3.0, 64-bit
17:28:59 2020-05-13 15:28:58.136 UTC [31564] LOG: listening on Unix socket "/tmp/gM2rQtsMib/.s.PGSQL.53078"
17:28:59 2020-05-13 15:28:58.139 UTC [31565] LOG: database system was shut down at 2020-05-13 15:28:58 UTC
17:28:59 2020-05-13 15:28:58.142 UTC [31564] LOG: database system is ready to accept connections
17:28:59 2020-05-13 15:28:58.230 UTC [31573] [unknown] LOG: LDAP login failed for user "?uid=test1" on server "localhost": Invalid DN syntax
17:28:59 2020-05-13 15:28:58.230 UTC [31573] [unknown] DETAIL: LDAP diagnostics: invalid DN
17:28:59 2020-05-13 15:28:58.230 UTC [31573] [unknown] FATAL: LDAP authentication failed for user "test1"
17:28:59 2020-05-13 15:28:58.230 UTC [31573] [unknown] DETAIL: Connection matched pg_hba.conf line 1: "local all all ldap ldapserver=localhost ldapport=53076 ldapprefix="?uid=" ldapsuffix="""
17:28:59 2020-05-13 15:28:58.264 UTC [31564] LOG: received fast shutdown request
17:28:59 2020-05-13 15:28:58.264 UTC [31564] LOG: aborting any active transactions
17:28:59 2020-05-13 15:28:58.266 UTC [31564] LOG: background worker "logical replication launcher" (PID 31571) exited with exit code 1
17:28:59 2020-05-13 15:28:58.266 UTC [31566] LOG: shutting down
17:28:59 2020-05-13 15:28:58.271 UTC [31564] LOG: database system is shut down
17:28:59 2020-05-13 15:28:58.384 UTC [31575] LOG: starting PostgreSQL 13devel (Debian 13~~devel~20200513.1505-1~801.git043e3e0.pgdg+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 9.3.0-12) 9.3.0, 64-bit
17:28:59 2020-05-13 15:28:58.385 UTC [31575] LOG: listening on Unix socket "/tmp/gM2rQtsMib/.s.PGSQL.53078"
17:28:59 2020-05-13 15:28:58.387 UTC [31576] LOG: database system was shut down at 2020-05-13 15:28:58 UTC
17:28:59 2020-05-13 15:28:58.390 UTC [31575] LOG: database system is ready to accept connections
17:28:59 2020-05-13 15:28:58.479 UTC [31584] [unknown] LOG: could not start LDAP TLS session: Connect error
17:28:59 2020-05-13 15:28:58.479 UTC [31584] [unknown] FATAL: LDAP authentication failed for user "test1"
17:28:59 2020-05-13 15:28:58.479 UTC [31584] [unknown] DETAIL: Connection matched pg_hba.conf line 1: "local all all ldap ldapserver=localhost ldapport=53076 ldapbasedn="dc=example,dc=net" ldapsearchfilter="(uid=$username)" ldaptls=1"
17:28:59 2020-05-13 15:28:58.514 UTC [31575] LOG: received fast shutdown request
17:28:59 2020-05-13 15:28:58.514 UTC [31575] LOG: aborting any active transactions
17:28:59 2020-05-13 15:28:58.516 UTC [31575] LOG: background worker "logical replication launcher" (PID 31582) exited with exit code 1
17:28:59 2020-05-13 15:28:58.516 UTC [31577] LOG: shutting down
17:28:59 2020-05-13 15:28:58.520 UTC [31575] LOG: database system is shut down
17:28:59 2020-05-13 15:28:58.634 UTC [31586] LOG: starting PostgreSQL 13devel (Debian 13~~devel~20200513.1505-1~801.git043e3e0.pgdg+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 9.3.0-12) 9.3.0, 64-bit
17:28:59 2020-05-13 15:28:58.634 UTC [31586] LOG: listening on Unix socket "/tmp/gM2rQtsMib/.s.PGSQL.53078"
17:28:59 2020-05-13 15:28:58.636 UTC [31587] LOG: database system was shut down at 2020-05-13 15:28:58 UTC
17:28:59 2020-05-13 15:28:58.639 UTC [31586] LOG: database system is ready to accept connections
17:28:59 2020-05-13 15:28:58.728 UTC [31595] [unknown] LOG: could not perform initial LDAP bind for ldapbinddn "" on server "localhost": Can't contact LDAP server
17:28:59 2020-05-13 15:28:58.728 UTC [31595] [unknown] FATAL: LDAP authentication failed for user "test1"
17:28:59 2020-05-13 15:28:58.728 UTC [31595] [unknown] DETAIL: Connection matched pg_hba.conf line 1: "local all all ldap ldapserver=localhost ldapscheme=ldaps ldapport=53077 ldapbasedn="dc=example,dc=net" ldapsearchfilter="(uid=$username)""
17:28:59 2020-05-13 15:28:58.763 UTC [31586] LOG: received fast shutdown request
17:28:59 2020-05-13 15:28:58.764 UTC [31586] LOG: aborting any active transactions
17:28:59 2020-05-13 15:28:58.766 UTC [31586] LOG: background worker "logical replication launcher" (PID 31593) exited with exit code 1
17:28:59 2020-05-13 15:28:58.766 UTC [31588] LOG: shutting down
17:28:59 2020-05-13 15:28:58.772 UTC [31586] LOG: database system is shut down
17:28:59 2020-05-13 15:28:58.886 UTC [31597] LOG: starting PostgreSQL 13devel (Debian 13~~devel~20200513.1505-1~801.git043e3e0.pgdg+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 9.3.0-12) 9.3.0, 64-bit
17:28:59 2020-05-13 15:28:58.886 UTC [31597] LOG: listening on Unix socket "/tmp/gM2rQtsMib/.s.PGSQL.53078"
17:28:59 2020-05-13 15:28:58.888 UTC [31598] LOG: database system was shut down at 2020-05-13 15:28:58 UTC
17:28:59 2020-05-13 15:28:58.891 UTC [31597] LOG: database system is ready to accept connections
17:28:59 2020-05-13 15:28:58.977 UTC [31606] [unknown] LOG: could not perform initial LDAP bind for ldapbinddn "" on server "localhost": Can't contact LDAP server
17:28:59 2020-05-13 15:28:58.977 UTC [31606] [unknown] FATAL: LDAP authentication failed for user "test1"
17:28:59 2020-05-13 15:28:58.977 UTC [31606] [unknown] DETAIL: Connection matched pg_hba.conf line 1: "local all all ldap ldapurl="ldaps://localhost:53077/dc=example,dc=net??sub?(uid=$username)""
17:28:59 2020-05-13 15:28:59.014 UTC [31597] LOG: received fast shutdown request
17:28:59 2020-05-13 15:28:59.014 UTC [31597] LOG: aborting any active transactions
17:28:59 2020-05-13 15:28:59.017 UTC [31597] LOG: background worker "logical replication launcher" (PID 31604) exited with exit code 1
17:28:59 2020-05-13 15:28:59.019 UTC [31599] LOG: shutting down
17:28:59 2020-05-13 15:28:59.025 UTC [31597] LOG: database system is shut down
17:28:59 2020-05-13 15:28:59.135 UTC [31608] LOG: starting PostgreSQL 13devel (Debian 13~~devel~20200513.1505-1~801.git043e3e0.pgdg+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 9.3.0-12) 9.3.0, 64-bit
17:28:59 2020-05-13 15:28:59.135 UTC [31608] LOG: listening on Unix socket "/tmp/gM2rQtsMib/.s.PGSQL.53078"
17:28:59 2020-05-13 15:28:59.137 UTC [31609] LOG: database system was shut down at 2020-05-13 15:28:59 UTC
17:28:59 2020-05-13 15:28:59.140 UTC [31608] LOG: database system is ready to accept connections
17:28:59 2020-05-13 15:28:59.228 UTC [31617] [unknown] LOG: could not start LDAP TLS session: Can't contact LDAP server
17:28:59 2020-05-13 15:28:59.228 UTC [31617] [unknown] FATAL: LDAP authentication failed for user "test1"
17:28:59 2020-05-13 15:28:59.228 UTC [31617] [unknown] DETAIL: Connection matched pg_hba.conf line 1: "local all all ldap ldapurl="ldaps://localhost:53077/dc=example,dc=net??sub?(uid=$username)" ldaptls=1"
17:28:59 2020-05-13 15:28:59.264 UTC [31608] LOG: received immediate shutdown request
17:28:59 2020-05-13 15:28:59.265 UTC [31613] WARNING: terminating connection because of crash of another server process
17:28:59 2020-05-13 15:28:59.265 UTC [31613] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
17:28:59 2020-05-13 15:28:59.265 UTC [31613] HINT: In a moment you should be able to reconnect to the database and repeat your command.
17:28:59 2020-05-13 15:28:59.268 UTC [31608] LOG: database system is shut down
17:28:59 ******** build/src/test/ldap/tmp_check/log/regress_log_001_auth ********
17:28:59 1..22
17:28:59 # Checking port 53076
17:28:59 # Found port 53076
17:28:59 # Checking port 53077
17:28:59 # Found port 53077
17:28:59 # setting up slapd
17:28:59 # Running: openssl req -new -nodes -keyout /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/ca.key -x509 -out /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/ca.crt -subj /CN=CA
17:28:59 Generating a RSA private key
17:28:59 ...........+++++
17:28:59 ...............+++++
17:28:59 writing new private key to '/<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/ca.key'
17:28:59 -----
17:28:59 # Running: openssl req -new -nodes -keyout /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/server.key -out /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/server.csr -subj /CN=server
17:28:59 Generating a RSA private key
17:28:59 ..............+++++
17:28:59 .............+++++
17:28:59 writing new private key to '/<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/server.key'
17:28:59 -----
17:28:59 # Running: openssl x509 -req -in /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/server.csr -CA /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/ca.crt -CAkey /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/ca.key -CAcreateserial -out /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd-certs/server.crt
17:28:59 Signature ok
17:28:59 subject=CN = server
17:28:59 Getting CA Private Key
17:28:59 # Running: /usr/sbin/slapd -f /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/slapd.conf -h ldap://localhost:53076 ldaps://localhost:53077
17:28:59 # Running: ldapsearch -h localhost -p 53076 -s base -b dc=example,dc=net -D cn=Manager,dc=example,dc=net -y /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/ldappassword -n 'objectclass=*'
17:28:59 # extended LDIF
17:28:59 #
17:28:59 # LDAPv3
17:28:59 # base <dc=example,dc=net> with scope baseObject
17:28:59 # filter: 'objectclass=*'
17:28:59 # requesting: ALL
17:28:59 #
17:28:59
17:28:59 # loading LDAP data
17:28:59 # Running: ldapadd -x -y /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/ldappassword -f authdata.ldif
17:28:59 adding new entry "dc=example,dc=net"
17:28:59
17:28:59 adding new entry "uid=test1,dc=example,dc=net"
17:28:59
17:28:59 adding new entry "uid=test2,dc=example,dc=net"
17:28:59
17:28:59 # Running: ldappasswd -x -y /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/ldappassword -s secret1 uid=test1,dc=example,dc=net
17:28:59 # Running: ldappasswd -x -y /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/ldappassword -s secret2 uid=test2,dc=example,dc=net
17:28:59 # setting up PostgreSQL instance
17:28:59 # Checking port 53078
17:28:59 # Found port 53078
17:28:59 Name: node
17:28:59 Data directory: /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata
17:28:59 Backup directory: /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/backup
17:28:59 Archive directory: /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/archives
17:28:59 Connection string: port=53078 host=/tmp/gM2rQtsMib
17:28:59 Log file: /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log
17:28:59 # Running: initdb -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -A trust -N
17:28:59 The files belonging to this database system will be owned by user "buildd".
17:28:59 This user must also own the server process.
17:28:59
17:28:59 The database cluster will be initialized with locale "C".
17:28:59 The default database encoding has accordingly been set to "SQL_ASCII".
17:28:59 The default text search configuration will be set to "english".
17:28:59
17:28:59 Data page checksums are disabled.
17:28:59
17:28:59 creating directory /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata ... ok
17:28:59 creating subdirectories ... ok
17:28:59 selecting dynamic shared memory implementation ... posix
17:28:59 selecting default max_connections ... 100
17:28:59 selecting default shared_buffers ... 128MB
17:28:59 selecting default time zone ... Etc/UTC
17:28:59 creating configuration files ... ok
17:28:59 running bootstrap script ... ok
17:28:59 performing post-bootstrap initialization ... ok
17:28:59
17:28:59 Sync to disk skipped.
17:28:59 The data directory might become corrupt if the operating system crashes.
17:28:59
17:28:59 Success. You can now start the database server using:
17:28:59
17:28:59 pg_ctl -D '/<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata' -l logfile start
17:28:59
17:28:59 # Running: /<<PKGBUILDDIR>>/build/src/test/ldap/../../../src/test/regress/pg_regress --config-auth /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata
17:28:59 ### Starting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log -o --cluster-name=node start
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31451
17:28:59 # running tests
17:28:59 # simple bind
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31466
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test0"
17:28:59 ok 1 - simple bind authentication fails if user not found in LDAP
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 ok 2 - simple bind authentication fails with wrong password
17:28:59 1
17:28:59 ok 3 - simple bind authentication succeeds
17:28:59 # search+bind
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31481
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test0"
17:28:59 ok 4 - search+bind authentication fails if user not found in LDAP
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 ok 5 - search+bind authentication fails with wrong password
17:28:59 1
17:28:59 ok 6 - search+bind authentication succeeds
17:28:59 # multiple servers
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31496
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test0"
17:28:59 ok 7 - search+bind authentication fails if user not found in LDAP
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 ok 8 - search+bind authentication fails with wrong password
17:28:59 1
17:28:59 ok 9 - search+bind authentication succeeds
17:28:59 # LDAP URLs
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31511
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test0"
17:28:59 ok 10 - search+bind with LDAP URL authentication fails if user not found in LDAP
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 ok 11 - search+bind with LDAP URL authentication fails with wrong password
17:28:59 1
17:28:59 ok 12 - search+bind with LDAP URL authentication succeeds
17:28:59 # search filters
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31526
17:28:59 1
17:28:59 ok 13 - search filter finds by uid
17:28:59 1
17:28:59 ok 14 - search filter finds by mail
17:28:59 # search filters in LDAP URLs
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31539
17:28:59 1
17:28:59 ok 15 - search filter finds by uid
17:28:59 1
17:28:59 ok 16 - search filter finds by mail
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31552
17:28:59 1
17:28:59 ok 17 - combined LDAP URL and search filter
17:28:59 # diagnostic message
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31564
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 ok 18 - any attempt fails due to bad search pattern
17:28:59 # TLS
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31575
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 not ok 19 - StartTLS
17:28:59
17:28:59 # Failed test 'StartTLS'
17:28:59 # at t/001_auth.pl line 169.
17:28:59 # got: '2'
17:28:59 # expected: '0'
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31586
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 not ok 20 - LDAPS
17:28:59
17:28:59 # Failed test 'LDAPS'
17:28:59 # at t/001_auth.pl line 169.
17:28:59 # got: '2'
17:28:59 # expected: '0'
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31597
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 not ok 21 - LDAPS with URL
17:28:59
17:28:59 # Failed test 'LDAPS with URL'
17:28:59 # at t/001_auth.pl line 169.
17:28:59 # got: '2'
17:28:59 # expected: '0'
17:28:59 ### Restarting node "node"
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -l /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/log/001_auth_node.log restart
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 waiting for server to start.... done
17:28:59 server started
17:28:59 # Postmaster PID for node "node" is 31608
17:28:59 psql: error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
17:28:59 ok 22 - bad combination of LDAPS and StartTLS
17:28:59 ### Stopping node "node" using mode immediate
17:28:59 # Running: pg_ctl -D /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata -m immediate stop
17:28:59 waiting for server to shut down.... done
17:28:59 server stopped
17:28:59 # No postmaster PID for node "node"
17:28:59 # Looks like you failed 3 tests of 22.
17:28:59 make[1]: *** [debian/rules:193: override_dh_auto_test-arch] Error 1
Christoph
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
@ 2020-05-14 04:47 ` Thomas Munro <[email protected]>
2020-05-14 11:10 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Thomas Munro @ 2020-05-14 04:47 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-hackers
On Thu, May 14, 2020 at 4:05 AM Christoph Berg <[email protected]> wrote:
> Some other problem emerged here in the ldap test:
Hi Christoph,
> 17:28:59 Data directory: /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata
I know nothing about Debian package building so I could be missing
something about how this works, but I wonder if our script variable
handling is hygienic enough for paths like that. Let's see... I get
an error due to that when I run "make -C src/test/ldap check" from a
source tree under "/<<PKGBUILDDIR>>/build":
sh: 1: cannot create /build/src/test/ldap/tmp_check/slapd.pid:
Directory nonexistent
That's fixable with:
- kill 'INT', `cat $slapd_pidfile` if -f $slapd_pidfile;
+ kill 'INT', `cat "$slapd_pidfile"` if -f "$slapd_pidfile";
That's a side issue, though.
> 17:28:59 # TLS
> 17:28:59 not ok 19 - StartTLS
> 17:28:59 not ok 20 - LDAPS
> 17:28:59 not ok 21 - LDAPS with URL
> It consistently fails on the build server, but works on my notebook.
> Maybe that simply means slapd is crashing, but there's no slapd
> output. Would it be possible to start slapd with "-d 255", even if
> that means it doesn't background itself?
That'd require more scripting to put it in the background...
> 17:28:59 2020-05-13 15:28:58.479 UTC [31584] [unknown] LOG: could not start LDAP TLS session: Connect error
> 17:28:59 2020-05-13 15:28:58.728 UTC [31595] [unknown] LOG: could not perform initial LDAP bind for ldapbinddn "" on server "localhost": Can't contact LDAP server
Hmm, I get exactly the same errors as this if I comment out the
following part of the test script:
# don't bother to check the server's cert (though perhaps we should)
append_to_file(
$ldap_conf,
qq{TLS_REQCERT never
});
That's a file that we point to with the environment variable LDAPCONF.
The man page for ldap.conf says:
Thus the following files and variables are read, in order:
variable $LDAPNOINIT, and if that is not set:
system file /etc/ldap/ldap.conf,
user files $HOME/ldaprc, $HOME/.ldaprc, ./ldaprc,
system file $LDAPCONF,
user files $HOME/$LDAPRC, $HOME/.$LDAPRC, ./$LDAPRC,
variables $LDAP<uppercase option name>.
Settings late in the list override earlier ones.
This leads me to suspect that something in your build server's
environment that comes later in that list is overridding the
TLS_REQCERT setting. If that's the explanation, perhaps we should do
this, which seems to work OK on my system, since it comes last in the
list:
-$ENV{'LDAPCONF'} = $ldap_conf;
+$ENV{'LDAPTLS_REQCERT'} = "never";
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
@ 2020-05-14 11:10 ` Christoph Berg <[email protected]>
2020-05-15 12:15 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Christoph Berg @ 2020-05-14 11:10 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
Re: Thomas Munro
> > 17:28:59 Data directory: /<<PKGBUILDDIR>>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata
>
> I know nothing about Debian package building so I could be missing
> something about how this works, but I wonder if our script variable
> handling is hygienic enough for paths like that.
That's from the sbuild log filter, the special chars are not the
problem:
17:06:41 I: NOTICE: Log filtering will replace 'build/postgresql-13-fdfISX/postgresql-13-13~~devel~20200513.1505' with '<<PKGBUILDDIR>>'
17:06:41 I: NOTICE: Log filtering will replace 'build/postgresql-13-fdfISX' with '<<BUILDDIR>>'
> > It consistently fails on the build server, but works on my notebook.
> > Maybe that simply means slapd is crashing, but there's no slapd
> > output. Would it be possible to start slapd with "-d 255", even if
> > that means it doesn't background itself?
>
> That'd require more scripting to put it in the background...
Maybe adding "&" is enough, provided it still writes the pid file for
shutting down...
> This leads me to suspect that something in your build server's
> environment that comes later in that list is overridding the
> TLS_REQCERT setting. If that's the explanation, perhaps we should do
> this, which seems to work OK on my system, since it comes last in the
> list:
It's not messing with environment variables in that area.
I'll see if I can catch a shell in the environment where it fails.
Christoph
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
2020-05-14 11:10 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
@ 2020-05-15 12:15 ` Christoph Berg <[email protected]>
2020-05-15 14:02 ` Re: ldap tls test fails in some environments Tom Lane <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Christoph Berg @ 2020-05-15 12:15 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; pgsql-hackers
> I'll see if I can catch a shell in the environment where it fails.
It failed right away when I tried on the buildd machine:
The slapd debug log is mostly garbage to me, the error seems to be
this:
ldap_read: want=8 error=Resource temporarily unavailable
src/test/ldap/t/001_auth.pl:
system_or_bail "sh", "-c", "$slapd -f $slapd_conf -h '$ldap_url $ldaps_url' -d 255 &";
END
{
kill 'INT', `cat $slapd_pidfile` if -f $slapd_pidfile;
}
tmp_check/log/001_auth_node.log:
2020-05-15 14:06:18.915 CEST [30486] [unknown] LOG: could not start LDAP TLS session: Connect error
2020-05-15 14:06:18.916 CEST [30486] [unknown] FATAL: LDAP authentication failed for user "test1"
2020-05-15 14:06:18.916 CEST [30486] [unknown] DETAIL: Connection matched pg_hba.conf line 1: "local all all ldap ldapserver=localhost ldapport=65510 ldapbasedn="dc=example,dc=net" ldapsearchfilter="(uid=$username)" ldaptls=1"
tmp_check/log/regress_log_001_auth:
# TLS
### Restarting node "node"
# Running: pg_ctl -D /home/myon/postgresql-13-13~~devel~20200515.0434/build/src/test/ldap/tmp_ch
eck/t_001_auth_node_data/pgdata -l /home/myon/postgresql-13-13~~devel~20200515.0434/build/src/te
st/ldap/tmp_check/log/001_auth_node.log restart
waiting for server to shut down.... done
server stopped
waiting for server to start.... done
server started
# Postmaster PID for node "node" is 30477
5ebe85ba daemon: activity on 1 descriptor
5ebe85ba daemon: activity on:
5ebe85ba slap_listener_activate(6):
5ebe85ba daemon: epoll: listen=6 busy
5ebe85ba daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ebe85ba >>> slap_listener(ldap://localhost:65510)
5ebe85ba daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ebe85ba daemon: accept() = 10
5ebe85ba daemon: listen=6, new connection on 10
5ebe85ba daemon: activity on 1 descriptor
5ebe85ba daemon: activity on:
5ebe85ba daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ebe85ba daemon: added 10r (active) listener=(nil)
5ebe85ba daemon: activity on 1 descriptor
5ebe85ba daemon: activity on: 10r
5ebe85ba daemon: read active on 10
5ebe85ba daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ebe85ba connection_get(10)
5ebe85ba connection_get(10): got connid=1033
5ebe85ba daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ebe85ba daemon: activity on 1 descriptor
5ebe85ba connection_read(10): checking for input on id=1033
ber_get_next
5ebe85ba daemon: activity on:
5ebe85ba daemon: epoll: listen=6 active_threads=0 tvp=NULL
ldap_read: want=8, got=8
0000: 30 1d 02 01 01 77 18 80 0....w..
ldap_read: want=23, got=23
5ebe85ba daemon: epoll: listen=7 active_threads=0 tvp=NULL
0000: 16 31 2e 33 2e 36 2e 31 2e 34 2e 31 2e 31 34 36 .1.3.6.1.4.1.146
5ebe85ba daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=9 active_threads=0 tvp=NULL
0010: 36 2e 32 30 30 33 37 6.20037
ber_get_next: tag 0x30 len 29 contents:
ber_dump: buf=0x7fa8ec107910 ptr=0x7fa8ec107910 end=0x7fa8ec10792d len=29
0000: 02 01 01 77 18 80 16 31 2e 33 2e 36 2e 31 2e 34 ...w...1.3.6.1.4
0010: 2e 31 2e 31 34 36 36 2e 32 30 30 33 37 .1.1466.20037
5ebe85ba op tag 0x77, time 1589544378
ber_get_next
ldap_read: want=8 error=Resource temporarily unavailable
5ebe85ba conn=1033 op=0 do_extended
ber_scanf fmt ({m) ber:
5ebe85ba daemon: activity on 1 descriptor
5ebe85ba daemon: activity on:ber_dump: buf=0x7fa8ec107910 ptr=0x7fa8ec107913 end=0x7fa8ec10792d len=26
0000: 77 18 80 16 31 2e 33 2e 36 2e 31 2e 34 2e 31 2e w...1.3.6.1.4.1.
0010: 31 34 36 36 2e 32 30 30 33 37 1466.20037
5ebe85ba daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ebe85ba do_extended: oid=1.3.6.1.4.1.1466.20037
5ebe85ba daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ebe85ba send_ldap_extended: err=0 oid= len=0
5ebe85ba send_ldap_response: msgid=1 tag=120 err=0
ber_flush2: 14 bytes to sd 10
0000: 30 0c 02 01 01 78 07 0a 01 00 04 00 04 00 0....x........
ldap_write: want=14, written=14
0000: 30 0c 02 01 01 78 07 0a 01 00 04 00 04 00 0....x........
5ebe85ba daemon: activity on 1 descriptor
5ebe85ba daemon: activity on: 10r
5ebe85ba daemon: read active on 10
5ebe85ba daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=7 active_threads=0 tvp=NULL
psql:5ebe85ba connection_get(10)
error: could not connect to server: FATAL: LDAP authentication failed for user "test1"
5ebe85ba daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ebe85ba connection_get(10): got connid=1033
5ebe85ba connection_read(10): checking for input on id=1033
tls_read: want=5, got=5
0000: 30 05 02 01 02 0....
TLS: can't accept: An unexpected TLS packet was received..
5ebe85ba connection_read(10): TLS accept failure error=-1 id=1033, closing
5ebe85ba connection_closing: readying conn=1033 sd=10 for close
5ebe85ba connection_close: conn=1033 sd=10
5ebe85ba daemon: removing 10
5ebe85ba daemon: activity on 1 descriptor
5ebe85ba daemon: activity on:
5ebe85ba daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ebe85ba daemon: epoll: listen=9 active_threads=0 tvp=NULL
not ok 19 - StartTLS
# Failed test 'StartTLS'
# at t/001_auth.pl line 169.
# got: '2'
# expected: '0'
### Restarting node "node"
Christoph
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
2020-05-14 11:10 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 12:15 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
@ 2020-05-15 14:02 ` Tom Lane <[email protected]>
2020-05-15 14:25 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Tom Lane @ 2020-05-15 14:02 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Christoph Berg <[email protected]> writes:
> The slapd debug log is mostly garbage to me, the error seems to be
> this:
> ldap_read: want=8 error=Resource temporarily unavailable
Hm, so EAGAIN (although that's a BSD-ish spelling of the strerror
string, which seems pretty odd in a Debian context). I don't think
that's actually an error, it's just the daemon's data collection
logic trying to read data that isn't there. It then goes on and
issues a response, so this must not indicate that the request is
incomplete --- it's just a useless speculative read.
Somebody should get out the LDAP RFCs and decode the packet contents
that this log helpfully provides, but I suspect that we're just looking
at an authentication failure; there's still not much clue as to why.
regards, tom lane
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
2020-05-14 11:10 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 12:15 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 14:02 ` Re: ldap tls test fails in some environments Tom Lane <[email protected]>
@ 2020-05-15 14:25 ` Christoph Berg <[email protected]>
2020-05-16 13:15 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Christoph Berg @ 2020-05-15 14:25 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Re: Tom Lane
> Somebody should get out the LDAP RFCs and decode the packet contents
> that this log helpfully provides, but I suspect that we're just looking
> at an authentication failure; there's still not much clue as to why.
The non-TLS tests work, so it's not a plain auth failure...
I'm attaching the full logs from that test, maybe someone with more
insight can compare the non-TLS with the TLS bits.
Would it help to re-run that with log_debug on the PG side?
Christoph
Attachments:
[application/gzip] 001_auth_node.log.gz (2.3K, ../../[email protected]/2-001_auth_node.log.gz)
download
[application/gzip] regress_log_001_auth.gz (35.4K, ../../[email protected]/3-regress_log_001_auth.gz)
download
[application/gzip] slapd.log.gz (4.8K, ../../[email protected]/4-slapd.log.gz)
download
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
2020-05-14 11:10 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 12:15 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 14:02 ` Re: ldap tls test fails in some environments Tom Lane <[email protected]>
2020-05-15 14:25 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
@ 2020-05-16 13:15 ` Thomas Munro <[email protected]>
2020-05-19 14:30 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
0 siblings, 1 reply; 278+ messages in thread
From: Thomas Munro @ 2020-05-16 13:15 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; Tom Lane <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On Sat, May 16, 2020 at 2:25 AM Christoph Berg <[email protected]> wrote:
> > Somebody should get out the LDAP RFCs and decode the packet contents
> > that this log helpfully provides, but I suspect that we're just looking
> > at an authentication failure; there's still not much clue as to why.
>
> The non-TLS tests work, so it's not a plain auth failure...
>
> I'm attaching the full logs from that test, maybe someone with more
> insight can compare the non-TLS with the TLS bits.
>
> Would it help to re-run that with log_debug on the PG side?
In your transcript for test 20, it looks like the client (PostgreSQL)
is hanging up without even sending a TLS ClientHello:
tls_read: want=5, got=0
...
psql:TLS: can't accept: The TLS connection was non-properly terminated..
With 001_auth.tl hacked to enable debug as you suggested*, on a local
Debian 10 system I see:
tls_read: want=5, got=5
0000: 16 03 01 01 4c ....L
That's: 0x16 = handshake record, 0x03 0x01 = TLS 1.0, and then a
record length for the following ClientHello. You're not even getting
that far, so I guess libdap is setting up the connection but then the
GNU TLS library (what Debian links libldap against) is not happy, even
before any negotiations begin. I wonder how to figure out why... does
this tell you anything?
$ENV{'LDAPCONF'} = $ldap_conf;
+$ENV{"GNUTLS_DEBUG_LEVEL"} = '99';
* Like this:
-system_or_bail $slapd, '-f', $slapd_conf, '-h', "$ldap_url $ldaps_url";
+system("$slapd -d 255 -f $slapd_conf -h '$ldap_url $ldaps_url' &");
^ permalink raw reply [nested|flat] 278+ messages in thread
* Re: ldap tls test fails in some environments
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Tom Lane <[email protected]>
2020-05-08 19:45 ` Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
2020-05-14 11:10 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 12:15 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 14:02 ` Re: ldap tls test fails in some environments Tom Lane <[email protected]>
2020-05-15 14:25 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-16 13:15 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
@ 2020-05-19 14:30 ` Christoph Berg <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Christoph Berg @ 2020-05-19 14:30 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Re: Thomas Munro
> In your transcript for test 20, it looks like the client (PostgreSQL)
> is hanging up without even sending a TLS ClientHello:
Maybe tests 19 and 20 are failing because 18 was already bad. (But
probably not.)
> I wonder how to figure out why... does this tell you anything?
>
> $ENV{'LDAPCONF'} = $ldap_conf;
> +$ENV{"GNUTLS_DEBUG_LEVEL"} = '99';
Sorry, I had not seen your message until now. Logs attached.
Even ldapsearch ... -ZZ is failing with that slapd config, so it's not
a PG problem.
$ GNUTLS_DEBUG_LEVEL=99 ldapsearch -h localhost -p 56118 -s base -b dc=example,dc=net -D cn=Manager,dc=example,dc=net -y /home/myon/postgresql-13-13~~devel~20200515.0434/build/src/test/ldap/tmp_check/ldappassword -n 'objectclass=*' -ZZ
gnutls[2]: Enabled GnuTLS 3.6.13 logging...
gnutls[2]: getrandom random generator was detected
gnutls[2]: Intel SSSE3 was detected
gnutls[2]: Intel AES accelerator was detected
gnutls[2]: Intel GCM accelerator was detected
gnutls[2]: cfg: unable to access: /etc/gnutls/config: 2
5ec3ec38 daemon: activity on 1 descriptor
5ec3ec38 daemon: activity on:
5ec3ec38 slap_listener_activate(6):
5ec3ec38 daemon: epoll: listen=6 busy
5ec3ec38 daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ec3ec38 >>> slap_listener(ldap://localhost:56118)
5ec3ec38 daemon: accept() = 10
5ec3ec38 daemon: listen=6, new connection on 10
5ec3ec38 daemon: activity on 1 descriptor
5ec3ec38 daemon: activity on:
5ec3ec38 daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ec3ec38 daemon: added 10r (active) listener=(nil)
5ec3ec38 daemon: activity on 1 descriptor
5ec3ec38 daemon: activity on: 10r
5ec3ec38 daemon: read active on 10
5ec3ec38 daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ec3ec38 connection_get(10)
5ec3ec38 daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ec3ec38 connection_get(10): got connid=1000
5ec3ec38 connection_read(10): checking for input on id=1000
5ec3ec38 daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ec3ec38 daemon: activity on 1 descriptor
ber_get_next
5ec3ec38 daemon: activity on:
ldap_read: want=8, got=8
0000: 30 1d 02 01 01 77 18 80 0....w..
5ec3ec38 daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=9 active_threads=0 tvp=NULL
ldap_read: want=23, got=23
0000: 16 31 2e 33 2e 36 2e 31 2e 34 2e 31 2e 31 34 36 .1.3.6.1.4.1.146
0010: 36 2e 32 30 30 33 37 6.20037
ber_get_next: tag 0x30 len 29 contents:
ber_dump: buf=0x7f0c50000bc0 ptr=0x7f0c50000bc0 end=0x7f0c50000bdd len=29
0000: 02 01 01 77 18 80 16 31 2e 33 2e 36 2e 31 2e 34 ...w...1.3.6.1.4
0010: 2e 31 2e 31 34 36 36 2e 32 30 30 33 37 .1.1466.20037
5ec3ec38 op tag 0x77, time 1589898296
ber_get_next
ldap_read: want=8 error=Resource temporarily unavailable
5ec3ec38 conn=1000 op=0 do_extended
5ec3ec38 daemon: activity on 1 descriptor
5ec3ec38 daemon: activity on:
ber_scanf fmt ({m) ber:
ber_dump: buf=0x7f0c50000bc0 ptr=0x7f0c50000bc3 end=0x7f0c50000bdd len=26
5ec3ec38 daemon: epoll: listen=6 active_threads=0 tvp=NULL
0000: 77 18 80 16 31 2e 33 2e 36 2e 31 2e 34 2e 31 2e w...1.3.6.1.4.1.
0010: 31 34 36 36 2e 32 30 30 33 37 1466.20037
5ec3ec38 daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ec3ec38 do_extended: oid=1.3.6.1.4.1.1466.20037
5ec3ec38 send_ldap_extended: err=0 oid= len=0
5ec3ec38 send_ldap_response: msgid=1 tag=120 err=0
ber_flush2: 14 bytes to sd 10
0000: 30 0c 02 01 01 78 07 0a 01 00 04 00 04 00 0....x........
ldap_write: want=14, written=14
0000: 30 0c 02 01 01 78 07 0a 01 00 04 00 04 00 0....x........
gnutls[2]: added 6 protocols, 29 ciphersuites, 19 sig algos and 10 groups into priority list
gnutls[3]: ASSERT: ../../../lib/x509/verify-high2.c[gnutls_x509_trust_list_add_trust_file]:361
ldap_start_tls: Connect error (-11)
5ec3ec38 daemon: activity on 1 descriptor
5ec3ec38 daemon: activity on: 10r
5ec3ec38 daemon: read active on 10
5ec3ec38 daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ec3ec38 connection_get(10)
5ec3ec38 connection_get(10): got connid=1000
5ec3ec38 daemon: epoll: listen=9 active_threads=0 tvp=NULL
5ec3ec38 connection_read(10): checking for input on id=1000
tls_read: want=5, got=5
0000: 30 05 02 01 02 0....
TLS: can't accept: An unexpected TLS packet was received..
5ec3ec38 connection_read(10): TLS accept failure error=-1 id=1000, closing
5ec3ec38 connection_closing: readying conn=1000 sd=10 for close
5ec3ec38 connection_close: conn=1000 sd=10
5ec3ec38 daemon: activity on 1 descriptor
5ec3ec38 daemon: removing 10
5ec3ec38 daemon: activity on:
5ec3ec38 daemon: epoll: listen=6 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=7 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=8 active_threads=0 tvp=NULL
5ec3ec38 daemon: epoll: listen=9 active_threads=0 tvp=NULL
There is an assertion failure:
gnutls[3]: ASSERT: ../../../lib/x509/verify-high2.c[gnutls_x509_trust_list_add_trust_file]:361
https://sources.debian.org/src/gnutls28/3.6.13-2/lib/x509/verify-high2.c/#L361
... which doesn't make sense to me yet.
Christoph
Attachments:
[application/gzip] ldap-debug.tar.gz (65.7K, ../../[email protected]/2-ldap-debug.tar.gz)
download
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 278+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 278+ messages in thread
end of thread, other threads:[~2025-07-26 00:57 UTC | newest]
Thread overview: 278+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-05-08 19:16 src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h Christoph Berg <[email protected]>
2020-05-08 19:34 ` Tom Lane <[email protected]>
2020-05-08 19:45 ` Christoph Berg <[email protected]>
2020-05-13 16:05 ` ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-14 04:47 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
2020-05-14 11:10 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 12:15 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-15 14:02 ` Re: ldap tls test fails in some environments Tom Lane <[email protected]>
2020-05-15 14:25 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2020-05-16 13:15 ` Re: ldap tls test fails in some environments Thomas Munro <[email protected]>
2020-05-19 14:30 ` Re: ldap tls test fails in some environments Christoph Berg <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox